[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:21:07 dist-test-slave-00qs krb5kdc[10297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:21:07 dist-test-slave-00qs krb5kdc[10297](info): set up 2 sockets
Sep 24 03:21:07 dist-test-slave-00qs krb5kdc[10297](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 03:21:08.357614 10290 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:21:08 dist-test-slave-00qs krb5kdc[10297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:21:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:21:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:21:08.557332 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:45427
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:45427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:08.966530 10310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:08.966971 10310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:08.967114 10310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:08.976526 10310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:21:08.976717 10310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:08.976799 10310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:08.976882 10310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:21:08.976943 10310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:21:08.989004 10310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:45427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:45427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:08.990737 10310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:08.992451 10310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:09.007332 10318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:09.007429 10315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:09.007437 10310 server_base.cc:1034] running on GCE node
W20260924 03:21:09.007296 10316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:09.008991 10310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:09.010771 10310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:09.012076 10310 hybrid_clock.cc:648] HybridClock initialized: now 1790220069011998 us; error 69 us; skew 500 ppm
I20260924 03:21:09.012550 10310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:09 dist-test-slave-00qs krb5kdc[10297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220069, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:09.016191 10310 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:21:09.017972 10310 webserver.cc:460] Webserver started at http://127.10.12.190:33255/ using document root <none> and password file <none>
I20260924 03:21:09.018812 10310 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:09.018939 10310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:09.019294 10310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:09.022754 10310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "cd96b5d0ab7b4664b9bed24f7ee8e048"
format_stamp: "Formatted at 2026-09-24 03:21:09 on dist-test-slave-00qs"
server_key: "cc7e3006f37cd7594b008174dcaeb617"
server_key_iv: "8079fb6c0edb6f65c83485ad30a709a6"
server_key_version: "encryptionkey@0"
I20260924 03:21:09.023633 10310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "cd96b5d0ab7b4664b9bed24f7ee8e048"
format_stamp: "Formatted at 2026-09-24 03:21:09 on dist-test-slave-00qs"
server_key: "cc7e3006f37cd7594b008174dcaeb617"
server_key_iv: "8079fb6c0edb6f65c83485ad30a709a6"
server_key_version: "encryptionkey@0"
I20260924 03:21:09.029953 10310 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 03:21:09.034231 10325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:09.036125 10310 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:21:09.036350 10310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "cd96b5d0ab7b4664b9bed24f7ee8e048"
format_stamp: "Formatted at 2026-09-24 03:21:09 on dist-test-slave-00qs"
server_key: "cc7e3006f37cd7594b008174dcaeb617"
server_key_iv: "8079fb6c0edb6f65c83485ad30a709a6"
server_key_version: "encryptionkey@0"
I20260924 03:21:09.036587 10310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:09.094334 10310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:09.099344 10310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:09.099670 10310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:09.119871 10310 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:45427
I20260924 03:21:09.119974 10376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:45427 every 8 connection(s)
I20260924 03:21:09.121480 10310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:21:09.125993 10377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:09.131980 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 10310
I20260924 03:21:09.132385 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:21:09.134997 10290 external_mini_cluster.cc:1428] Setting key e6541a2cd956fd73612aab5ef6849c3d
I20260924 03:21:09.139921 10377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd96b5d0ab7b4664b9bed24f7ee8e048: Bootstrap starting.
I20260924 03:21:09.144142 10377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd96b5d0ab7b4664b9bed24f7ee8e048: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:09.145361 10377 log.cc:824] T 00000000000000000000000000000000 P cd96b5d0ab7b4664b9bed24f7ee8e048: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:09.148572 10377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd96b5d0ab7b4664b9bed24f7ee8e048: No bootstrap required, opened a new log
I20260924 03:21:09.158882 10377 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd96b5d0ab7b4664b9bed24f7ee8e048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd96b5d0ab7b4664b9bed24f7ee8e048" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45427 } }
I20260924 03:21:09.159482 10377 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd96b5d0ab7b4664b9bed24f7ee8e048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:09.159603 10377 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd96b5d0ab7b4664b9bed24f7ee8e048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd96b5d0ab7b4664b9bed24f7ee8e048, State: Initialized, Role: FOLLOWER
I20260924 03:21:09.160491 10377 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd96b5d0ab7b4664b9bed24f7ee8e048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd96b5d0ab7b4664b9bed24f7ee8e048" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45427 } }
I20260924 03:21:09.160825 10377 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd96b5d0ab7b4664b9bed24f7ee8e048 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:21:09.160969 10377 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd96b5d0ab7b4664b9bed24f7ee8e048 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:21:09.161164 10377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd96b5d0ab7b4664b9bed24f7ee8e048 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:09.163720 10377 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd96b5d0ab7b4664b9bed24f7ee8e048 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd96b5d0ab7b4664b9bed24f7ee8e048" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45427 } }
I20260924 03:21:09.164516 10377 leader_election.cc:302] T 00000000000000000000000000000000 P cd96b5d0ab7b4664b9bed24f7ee8e048 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd96b5d0ab7b4664b9bed24f7ee8e048; no voters:
I20260924 03:21:09.165195 10377 leader_election.cc:288] T 00000000000000000000000000000000 P cd96b5d0ab7b4664b9bed24f7ee8e048 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:21:09.165395 10382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd96b5d0ab7b4664b9bed24f7ee8e048 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:09.166854 10382 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd96b5d0ab7b4664b9bed24f7ee8e048 [term 1 LEADER]: Becoming Leader. State: Replica: cd96b5d0ab7b4664b9bed24f7ee8e048, State: Running, Role: LEADER
I20260924 03:21:09.167620 10377 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd96b5d0ab7b4664b9bed24f7ee8e048 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:21:09.167546 10382 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd96b5d0ab7b4664b9bed24f7ee8e048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd96b5d0ab7b4664b9bed24f7ee8e048" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45427 } }
Sep 24 03:21:09 dist-test-slave-00qs krb5kdc[10297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:09.171972 10383 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd96b5d0ab7b4664b9bed24f7ee8e048 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd96b5d0ab7b4664b9bed24f7ee8e048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd96b5d0ab7b4664b9bed24f7ee8e048" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45427 } } }
I20260924 03:21:09.172746 10383 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd96b5d0ab7b4664b9bed24f7ee8e048 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:09.173342 10384 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd96b5d0ab7b4664b9bed24f7ee8e048 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd96b5d0ab7b4664b9bed24f7ee8e048. Latest consensus state: current_term: 1 leader_uuid: "cd96b5d0ab7b4664b9bed24f7ee8e048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd96b5d0ab7b4664b9bed24f7ee8e048" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45427 } } }
I20260924 03:21:09.173637 10384 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd96b5d0ab7b4664b9bed24f7ee8e048 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:09.186138 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:09.140349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41400 (local address 127.10.12.190:45427)
0924 03:21:09.141642 (+ 1293us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:09.141661 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:09.142969 (+ 1308us) server_negotiation.cc:407] Connection header received
0924 03:21:09.154177 (+ 11208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:09.154218 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:09.154931 (+ 713us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:09.155489 (+ 558us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:09.158958 (+ 3469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:09.161015 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:09.163044 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:09.164244 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:09.164460 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:09.173088 (+ 8628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:09.173104 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:09.173127 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:09.173184 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:09.178033 (+ 4849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:09.179495 (+ 1462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:09.179508 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:09.179519 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:09.179835 (+ 316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:09.180828 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:09.180835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:09.180839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:09.181785 (+ 946us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:09.182124 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:09.183741 (+ 1617us) server_negotiation.cc:299] Negotiation successful
0924 03:21:09.184368 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":464,"threads_started":1}
I20260924 03:21:09.190613 10391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:21:09.196869 10391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:21:09.208211 10391 catalog_manager.cc:1347] Generated new cluster ID: 2bd463d4ce0f4e449dc82eb8d331836b
I20260924 03:21:09.208350 10391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:21:09.217011 10391 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:21:09.219149 10391 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:21:09.231473 10391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd96b5d0ab7b4664b9bed24f7ee8e048: Generated new TSK 0
I20260924 03:21:09.232893 10391 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
I20260924 03:21:09.284684 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:45427
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:09.661213 10403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:09.661571 10403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:09.661701 10403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:09.669710 10403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:09.669875 10403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:09.670042 10403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:21:09.680745 10403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:45427
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:09.682130 10403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:09.683668 10403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:09.697520 10409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:09.697546 10411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:09.697608 10408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:09.701836 10403 server_base.cc:1034] running on GCE node
I20260924 03:21:09.702575 10403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:09.703809 10403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:09.705057 10403 hybrid_clock.cc:648] HybridClock initialized: now 1790220069704984 us; error 65 us; skew 500 ppm
I20260924 03:21:09.705415 10403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:09 dist-test-slave-00qs krb5kdc[10297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220069, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:09.708804 10403 init.cc:377] Logged in from keytab as kudu/127.10.12.129@KRBTEST.COM (short username kudu)
I20260924 03:21:09.710590 10403 webserver.cc:460] Webserver started at http://127.10.12.129:34111/ using document root <none> and password file <none>
I20260924 03:21:09.711414 10403 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:09.711544 10403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:09.711861 10403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:09.715113 10403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "093fbf6d5055425784b7c82b28f61c85"
format_stamp: "Formatted at 2026-09-24 03:21:09 on dist-test-slave-00qs"
server_key: "14c2ac5394ab1dfbd22441837a2f7719"
server_key_iv: "6ac7ff144db227890fcd5f974bb83bde"
server_key_version: "encryptionkey@0"
I20260924 03:21:09.716019 10403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "093fbf6d5055425784b7c82b28f61c85"
format_stamp: "Formatted at 2026-09-24 03:21:09 on dist-test-slave-00qs"
server_key: "14c2ac5394ab1dfbd22441837a2f7719"
server_key_iv: "6ac7ff144db227890fcd5f974bb83bde"
server_key_version: "encryptionkey@0"
I20260924 03:21:09.722218 10403 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 03:21:09.726521 10418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:09.728852 10403 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:21:09.729082 10403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "093fbf6d5055425784b7c82b28f61c85"
format_stamp: "Formatted at 2026-09-24 03:21:09 on dist-test-slave-00qs"
server_key: "14c2ac5394ab1dfbd22441837a2f7719"
server_key_iv: "6ac7ff144db227890fcd5f974bb83bde"
server_key_version: "encryptionkey@0"
I20260924 03:21:09.729324 10403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:09.748956 10403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:09.754758 10403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:09.755064 10403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:09.756222 10403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:21:09.757752 10403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:09.757843 10403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:09.757982 10403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:09.758035 10403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:09.791671 10403 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:41705
I20260924 03:21:09.791689 10530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:41705 every 8 connection(s)
I20260924 03:21:09.793306 10403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:21:09.795353 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 10403
I20260924 03:21:09.795579 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:21:09.796291 10290 external_mini_cluster.cc:1428] Setting key 3ee88679be8137d1f80e6ba950055d33
Sep 24 03:21:09 dist-test-slave-00qs krb5kdc[10297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220069, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.130@KRBTEST.COM" created.
I20260924 03:21:09.819555 10533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:09.796185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:58973 (local address 127.10.12.190:45427)
0924 03:21:09.798791 (+ 2606us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:09.798797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:09.798815 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:21:09.801498 (+ 2683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:09.801502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:09.801581 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:09.801700 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:09.804019 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:09.805090 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:09.806887 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:09.807472 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:09.807617 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:09.811908 (+ 4291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:09.811917 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:09.811923 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:09.812004 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:09.815016 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:09.816224 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:09.816232 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:09.816247 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:09.816602 (+ 355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:09.817321 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:09.817330 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:09.817334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:09.817868 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:09.818071 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:09.818914 (+ 843us) server_negotiation.cc:299] Negotiation successful
0924 03:21:09.819189 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2505,"thread_start_us":2378,"threads_started":1}
I20260924 03:21:09.820514 10534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:09.796928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:45427 (local address 127.10.12.129:58973)
0924 03:21:09.797962 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:09.798025 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:09.801063 (+ 3038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:09.802038 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:09.802053 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:09.802867 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:09.803718 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:09.803746 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:09.805322 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:09.805327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:09.806686 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:09.806701 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:09.807748 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:09.807754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:09.808024 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:09.808863 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:09.808893 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:09.811576 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:09.815226 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:09.815236 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:09.815253 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:09.815924 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:09.816812 (+ 888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:09.816817 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:09.816821 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:09.817079 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:09.818056 (+ 977us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:09.818063 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:09.818463 (+ 400us) client_negotiation.cc:769] Sending connection context
0924 03:21:09.818731 (+ 268us) client_negotiation.cc:240] Negotiation successful
0924 03:21:09.819247 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":393,"threads_started":1}
I20260924 03:21:09.822232 10531 heartbeater.cc:347] Connected to a master server at 127.10.12.190:45427
I20260924 03:21:09.822553 10531 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:09.823339 10531 heartbeater.cc:511] Master 127.10.12.190:45427 requested a full tablet report, sending...
I20260924 03:21:09.825634 10342 ts_manager.cc:194] Registered new tserver with Master: 093fbf6d5055425784b7c82b28f61c85 (127.10.12.129:41705)
I20260924 03:21:09.827214 10342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.129@KRBTEST.COM'} at 127.10.12.129:58973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
I20260924 03:21:09.840555 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:45427
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:10.185153 10537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:10.185487 10537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:10.185632 10537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:10.193600 10537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:10.193778 10537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:10.194087 10537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:21:10.204703 10537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:45427
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:10.206076 10537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:10.207463 10537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:10.221956 10543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:10.222000 10542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:10.222718 10545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:10.223032 10537 server_base.cc:1034] running on GCE node
I20260924 03:21:10.223881 10537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:10.225844 10537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:10.227422 10537 hybrid_clock.cc:648] HybridClock initialized: now 1790220070227372 us; error 53 us; skew 500 ppm
I20260924 03:21:10.227941 10537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:10 dist-test-slave-00qs krb5kdc[10297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220070, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:10.233047 10537 init.cc:377] Logged in from keytab as kudu/127.10.12.130@KRBTEST.COM (short username kudu)
I20260924 03:21:10.235198 10537 webserver.cc:460] Webserver started at http://127.10.12.130:39145/ using document root <none> and password file <none>
I20260924 03:21:10.236349 10537 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:10.236521 10537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:10.236920 10537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:10.241590 10537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "482cf0d0e3794db28f09cbd744c619f7"
format_stamp: "Formatted at 2026-09-24 03:21:10 on dist-test-slave-00qs"
server_key: "ef86498721629406a413253dfdf7946a"
server_key_iv: "402c16b321d18777567706603b8c8ef5"
server_key_version: "encryptionkey@0"
I20260924 03:21:10.242749 10537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "482cf0d0e3794db28f09cbd744c619f7"
format_stamp: "Formatted at 2026-09-24 03:21:10 on dist-test-slave-00qs"
server_key: "ef86498721629406a413253dfdf7946a"
server_key_iv: "402c16b321d18777567706603b8c8ef5"
server_key_version: "encryptionkey@0"
I20260924 03:21:10.251335 10537 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260924 03:21:10.257268 10552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:10.259776 10537 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:21:10.260108 10537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "482cf0d0e3794db28f09cbd744c619f7"
format_stamp: "Formatted at 2026-09-24 03:21:10 on dist-test-slave-00qs"
server_key: "ef86498721629406a413253dfdf7946a"
server_key_iv: "402c16b321d18777567706603b8c8ef5"
server_key_version: "encryptionkey@0"
I20260924 03:21:10.260425 10537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:10.297958 10537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:10.305312 10537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:10.305712 10537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:10.307360 10537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:21:10.309491 10537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:10.309624 10537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:10.309762 10537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:10.309828 10537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:10.349721 10537 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:45017
I20260924 03:21:10.349736 10664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:45017 every 8 connection(s)
I20260924 03:21:10.351569 10537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:21:10.359844 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 10537
I20260924 03:21:10.360080 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:21:10.360639 10290 external_mini_cluster.cc:1428] Setting key c5ac63ad0b48be2c8e390f17d7ddbe40
Sep 24 03:21:10 dist-test-slave-00qs krb5kdc[10297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220070, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:10.378367 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:10.355127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:45427 (local address 127.10.12.130:49503)
0924 03:21:10.356102 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:10.356153 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:10.358065 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:10.358851 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:10.358861 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:10.359461 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:10.360393 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:10.360423 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:10.362046 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:10.362052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:10.363612 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:10.363628 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:10.364500 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:10.364507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:10.364746 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:10.365416 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:10.365441 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:10.368097 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:10.372550 (+ 4453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:10.372562 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:10.372585 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:10.373287 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:10.373887 (+ 600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:10.373892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:10.373896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:10.374193 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:10.375079 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:10.375089 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:10.375683 (+ 594us) client_negotiation.cc:769] Sending connection context
0924 03:21:10.376071 (+ 388us) client_negotiation.cc:240] Negotiation successful
0924 03:21:10.376759 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":406,"threads_started":1}
I20260924 03:21:10.379781 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:10.354533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:49503 (local address 127.10.12.190:45427)
0924 03:21:10.356929 (+ 2396us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:10.356936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:10.356961 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:21:10.358380 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:10.358385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:10.358472 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:10.358570 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:10.360661 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:10.361837 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:10.363785 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:10.364264 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:10.364381 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:10.368444 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:10.368452 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:10.368457 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:10.368503 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:10.372309 (+ 3806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:10.373461 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:10.373467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:10.373471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:10.373684 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:10.374354 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:10.374360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:10.374363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:10.374845 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:10.374993 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:10.379063 (+ 4070us) server_negotiation.cc:299] Negotiation successful
0924 03:21:10.379369 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2266,"thread_start_us":2143,"threads_started":1}
WARNING: no policy specified for kudu/127.10.12.131@KRBTEST.COM; defaulting to no policy
I20260924 03:21:10.381145 10665 heartbeater.cc:347] Connected to a master server at 127.10.12.190:45427
I20260924 03:21:10.381599 10665 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.131@KRBTEST.COM" created.
I20260924 03:21:10.382614 10665 heartbeater.cc:511] Master 127.10.12.190:45427 requested a full tablet report, sending...
I20260924 03:21:10.385035 10342 ts_manager.cc:194] Registered new tserver with Master: 482cf0d0e3794db28f09cbd744c619f7 (127.10.12.130:45017)
I20260924 03:21:10.386150 10342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.130@KRBTEST.COM'} at 127.10.12.130:49503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
I20260924 03:21:10.408574 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:45427
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:10.750805 10671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:10.751113 10671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:10.751236 10671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:10.759312 10671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:10.759449 10671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:10.759605 10671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:21:10.770301 10671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:45427
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:10.771796 10671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:10.773156 10671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:10.787710 10676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:10.787662 10679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:10.787725 10677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:10.788688 10671 server_base.cc:1034] running on GCE node
I20260924 03:21:10.789321 10671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:10.790336 10671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:10.791568 10671 hybrid_clock.cc:648] HybridClock initialized: now 1790220070791508 us; error 53 us; skew 500 ppm
I20260924 03:21:10.791993 10671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:10 dist-test-slave-00qs krb5kdc[10297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220070, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:10.795643 10671 init.cc:377] Logged in from keytab as kudu/127.10.12.131@KRBTEST.COM (short username kudu)
I20260924 03:21:10.798127 10671 webserver.cc:460] Webserver started at http://127.10.12.131:42593/ using document root <none> and password file <none>
I20260924 03:21:10.798923 10671 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:10.799059 10671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:10.799368 10671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:10.802431 10671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "85f692d561b043f1bf5b8be0c3722a78"
format_stamp: "Formatted at 2026-09-24 03:21:10 on dist-test-slave-00qs"
server_key: "ce866f4c13932c14901ba8f312a8f05c"
server_key_iv: "8bc18429dae90d50e8c8d14dfdb3e8d0"
server_key_version: "encryptionkey@0"
I20260924 03:21:10.803243 10671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "85f692d561b043f1bf5b8be0c3722a78"
format_stamp: "Formatted at 2026-09-24 03:21:10 on dist-test-slave-00qs"
server_key: "ce866f4c13932c14901ba8f312a8f05c"
server_key_iv: "8bc18429dae90d50e8c8d14dfdb3e8d0"
server_key_version: "encryptionkey@0"
I20260924 03:21:10.808957 10671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260924 03:21:10.813030 10686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:10.814590 10671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:21:10.814836 10671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "85f692d561b043f1bf5b8be0c3722a78"
format_stamp: "Formatted at 2026-09-24 03:21:10 on dist-test-slave-00qs"
server_key: "ce866f4c13932c14901ba8f312a8f05c"
server_key_iv: "8bc18429dae90d50e8c8d14dfdb3e8d0"
server_key_version: "encryptionkey@0"
I20260924 03:21:10.815060 10671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:10.830560 10531 heartbeater.cc:503] Master 127.10.12.190:45427 was elected leader, sending a full tablet report...
I20260924 03:21:10.836771 10671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:10.841607 10671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:10.841938 10671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:10.842955 10671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:21:10.844413 10671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:10.844492 10671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:10.844619 10671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:10.844681 10671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:10.883157 10671 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:43023
I20260924 03:21:10.883188 10798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:43023 every 8 connection(s)
I20260924 03:21:10.884579 10671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:21:10.887403 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 10671
I20260924 03:21:10.887591 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:21:10.888170 10290 external_mini_cluster.cc:1428] Setting key e4ac456639b9063eba3182d93882da76
Sep 24 03:21:10 dist-test-slave-00qs krb5kdc[10297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220070, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:10.907068 10801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:10.887650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:39661 (local address 127.10.12.190:45427)
0924 03:21:10.888244 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:10.888251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:10.889311 (+ 1060us) server_negotiation.cc:407] Connection header received
0924 03:21:10.891193 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:10.891199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:10.891286 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:10.891389 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:10.893058 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:10.893862 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:10.895483 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:10.895878 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:10.896010 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:10.899724 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:10.899731 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:10.899735 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:10.899774 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:10.902456 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:10.903973 (+ 1517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:10.903981 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:10.903985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:10.904208 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:10.905140 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:10.905146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:10.905149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:10.905503 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:10.905643 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:10.906507 (+ 864us) server_negotiation.cc:299] Negotiation successful
0924 03:21:10.906783 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":307,"threads_started":1}
I20260924 03:21:10.907924 10802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:10.888324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:45427 (local address 127.10.12.131:39661)
0924 03:21:10.889140 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:10.889192 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:10.890898 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:10.891643 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:10.891652 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:10.892267 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:10.892868 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:10.892888 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:10.894031 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:10.894035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:10.895315 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:10.895330 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:10.896132 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:10.896139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:10.896386 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:10.897044 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:10.897072 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:10.899412 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:10.902793 (+ 3381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:10.902809 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:10.902834 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:10.903707 (+ 873us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:10.904420 (+ 713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:10.904427 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:10.904432 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:10.904925 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:10.905690 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:10.905698 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:10.906083 (+ 385us) client_negotiation.cc:769] Sending connection context
0924 03:21:10.906356 (+ 273us) client_negotiation.cc:240] Negotiation successful
0924 03:21:10.906805 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":304,"threads_started":1}
I20260924 03:21:10.910089 10799 heartbeater.cc:347] Connected to a master server at 127.10.12.190:45427
I20260924 03:21:10.910487 10799 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:10.911346 10799 heartbeater.cc:511] Master 127.10.12.190:45427 requested a full tablet report, sending...
I20260924 03:21:10.913262 10342 ts_manager.cc:194] Registered new tserver with Master: 85f692d561b043f1bf5b8be0c3722a78 (127.10.12.131:43023)
I20260924 03:21:10.914554 10342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.131@KRBTEST.COM'} at 127.10.12.131:39661
I20260924 03:21:10.918823 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 03:21:10 dist-test-slave-00qs krb5kdc[10297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220070, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 03:21:10 dist-test-slave-00qs krb5kdc[10297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220070, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.12.129@KRBTEST.COM
I20260924 03:21:10.964670 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:10.947614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49812 (local address 127.10.12.129:41705)
0924 03:21:10.948315 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:10.948324 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:10.948361 (+ 37us) server_negotiation.cc:407] Connection header received
0924 03:21:10.948516 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:10.948524 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:10.948767 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:10.948913 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:10.950444 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:10.951478 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:10.952848 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:10.953468 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:10.953594 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:10.958099 (+ 4505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:10.958116 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:10.958133 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:10.958185 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:10.961588 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:10.962511 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:10.962519 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:10.962522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:10.962676 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:10.963093 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:10.963097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:10.963100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:10.963492 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:10.963733 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:10.964061 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 03:21:10.964299 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":352,"threads_started":1}
W20260924 03:21:10.966001 10446 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49812
Sep 24 03:21:10 dist-test-slave-00qs krb5kdc[10297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 03:21:10 dist-test-slave-00qs krb5kdc[10297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.129@KRBTEST.COM
I20260924 03:21:11.009667 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:10.994451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49814 (local address 127.10.12.129:41705)
0924 03:21:10.994661 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:10.994668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:10.995019 (+ 351us) server_negotiation.cc:407] Connection header received
0924 03:21:10.995202 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:10.995206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:10.995275 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:10.995360 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:10.996788 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:10.997500 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:10.998777 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:10.999195 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:10.999305 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:11.003561 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:11.003568 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:11.003573 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:11.003618 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:11.007033 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:11.007764 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:11.007769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:11.007771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:11.007911 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:11.008303 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:11.008308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:11.008310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:11.008575 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:11.008817 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:11.009114 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 03:21:11.009348 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 03:21:11.011286 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 10403
I20260924 03:21:11.024384 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:11.012386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49830 (local address 127.10.12.129:41705)
0924 03:21:11.012590 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:11.012596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:11.012834 (+ 238us) server_negotiation.cc:407] Connection header received
0924 03:21:11.012985 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:11.012988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:11.013049 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:11.013127 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:11.014388 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:11.015079 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:11.016427 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:11.016853 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:11.016964 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:11.019343 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:11.019348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:11.019352 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:11.019387 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:11.021812 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:11.022478 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:11.022482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:11.022485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:11.022626 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:11.022986 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:11.022991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:11.022993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:11.023261 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:11.023480 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:11.023780 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 03:21:11.024046 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 03:21:11.025162 10526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:11.108834 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 10403
I20260924 03:21:11.124097 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 10537
Sep 24 03:21:11 dist-test-slave-00qs krb5kdc[10297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.130@KRBTEST.COM
I20260924 03:21:11.141003 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:11.125031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43962 (local address 127.10.12.130:45017)
0924 03:21:11.125541 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:11.125549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:11.125573 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:21:11.125639 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:11.125645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:11.125869 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:11.126012 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:11.127275 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:11.128359 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:11.129629 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:11.130202 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:11.130334 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:11.134762 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:11.134777 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:11.134793 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:11.134835 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:11.137915 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:11.138691 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:11.138704 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:11.138707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:11.138846 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:11.139269 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:11.139273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:11.139275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:11.139641 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:11.139987 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:11.140296 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 03:21:11.140579 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":261,"threads_started":1}
I20260924 03:21:11.142349 10660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:11.228041 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 10537
I20260924 03:21:11.241691 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 10671
Sep 24 03:21:11 dist-test-slave-00qs krb5kdc[10297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.131@KRBTEST.COM
I20260924 03:21:11.259927 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:11.242774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50974 (local address 127.10.12.131:43023)
0924 03:21:11.243442 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:11.243450 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:11.243478 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:21:11.243547 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:11.243554 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:11.243770 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:11.243910 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:11.245284 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:11.246361 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:11.247847 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:11.248714 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:11.248882 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:11.253696 (+ 4814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:11.253715 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:11.253733 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:11.253795 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:11.256874 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:11.257676 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:11.257683 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:11.257686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:11.257845 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:11.258238 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:11.258242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:11.258244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:11.258633 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:11.258960 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:11.259262 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 03:21:11.259553 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":306,"threads_started":1}
I20260924 03:21:11.261376 10794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:11.345705 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 10671
I20260924 03:21:11.358018 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 10310
I20260924 03:21:11.359247 10372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:11.468696 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 10310
2026-09-24T03:21:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4167 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:21:11 dist-test-slave-00qs krb5kdc[10821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:21:11 dist-test-slave-00qs krb5kdc[10821](info): set up 2 sockets
Sep 24 03:21:11 dist-test-slave-00qs krb5kdc[10821](info): commencing operation
krb5kdc: starting...
W20260924 03:21:12.496109 10290 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.006s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:21:12 dist-test-slave-00qs krb5kdc[10821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:21:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:21:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:21:12.677917 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:36365
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:44711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:36365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:13.013722 10834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:13.014056 10834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:13.014176 10834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:13.021746 10834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:21:13.021880 10834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:13.021927 10834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:13.022001 10834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:21:13.022060 10834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:21:13.032454 10834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:44711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:36365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:36365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:13.033911 10834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:13.035312 10834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:13.048034 10839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:13.048098 10842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:13.048059 10840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:13.049176 10834 server_base.cc:1034] running on GCE node
I20260924 03:21:13.050141 10834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:13.051790 10834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:13.053021 10834 hybrid_clock.cc:648] HybridClock initialized: now 1790220073052952 us; error 61 us; skew 500 ppm
I20260924 03:21:13.053426 10834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:13 dist-test-slave-00qs krb5kdc[10821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220073, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:13.066712 10834 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:21:13.068496 10834 webserver.cc:460] Webserver started at http://127.10.12.190:45429/ using document root <none> and password file <none>
I20260924 03:21:13.069224 10834 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:13.069350 10834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:13.069629 10834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:13.072887 10834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "bbaac6feda8d49139edfaec138ccfa02"
format_stamp: "Formatted at 2026-09-24 03:21:13 on dist-test-slave-00qs"
server_key: "060a4443c22e21d18c5fff8723594be8"
server_key_iv: "6f0253893ce765f56de876dd23c36eff"
server_key_version: "encryptionkey@0"
I20260924 03:21:13.073675 10834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "bbaac6feda8d49139edfaec138ccfa02"
format_stamp: "Formatted at 2026-09-24 03:21:13 on dist-test-slave-00qs"
server_key: "060a4443c22e21d18c5fff8723594be8"
server_key_iv: "6f0253893ce765f56de876dd23c36eff"
server_key_version: "encryptionkey@0"
I20260924 03:21:13.079362 10834 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 03:21:13.083336 10849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:13.085109 10834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:21:13.085289 10834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "bbaac6feda8d49139edfaec138ccfa02"
format_stamp: "Formatted at 2026-09-24 03:21:13 on dist-test-slave-00qs"
server_key: "060a4443c22e21d18c5fff8723594be8"
server_key_iv: "6f0253893ce765f56de876dd23c36eff"
server_key_version: "encryptionkey@0"
I20260924 03:21:13.085490 10834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:13.118348 10834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:13.123215 10834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:13.123502 10834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:13.142571 10834 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:36365
I20260924 03:21:13.142578 10900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:36365 every 8 connection(s)
I20260924 03:21:13.144074 10834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:21:13.146575 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 10834
I20260924 03:21:13.146823 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:21:13.147578 10290 external_mini_cluster.cc:1428] Setting key 2c206e69e8040bfba675d5ad097361c2
I20260924 03:21:13.148823 10901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 03:21:13 dist-test-slave-00qs krb5kdc[10821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:13.162166 10901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbaac6feda8d49139edfaec138ccfa02: Bootstrap starting.
I20260924 03:21:13.166015 10901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbaac6feda8d49139edfaec138ccfa02: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:13.167632 10901 log.cc:824] T 00000000000000000000000000000000 P bbaac6feda8d49139edfaec138ccfa02: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:13.171653 10901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbaac6feda8d49139edfaec138ccfa02: No bootstrap required, opened a new log
I20260924 03:21:13.172801 10904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:13.149781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40038 (local address 127.10.12.190:36365)
0924 03:21:13.150735 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:13.150749 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:13.150796 (+ 47us) server_negotiation.cc:407] Connection header received
0924 03:21:13.152496 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:13.152521 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:13.152980 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:13.153408 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:13.154936 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:13.156337 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:13.157594 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:13.158343 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:13.158593 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:13.163003 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:13.163011 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:13.163029 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:13.163080 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:13.166794 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:13.168140 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:13.168149 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:13.168158 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:13.168438 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:13.169025 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:13.169030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:13.169032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:13.169873 (+ 841us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:13.170197 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:13.170776 (+ 579us) server_negotiation.cc:299] Negotiation successful
0924 03:21:13.171304 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":372,"threads_started":1}
I20260924 03:21:13.180714 10901 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbaac6feda8d49139edfaec138ccfa02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbaac6feda8d49139edfaec138ccfa02" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36365 } }
I20260924 03:21:13.181133 10901 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbaac6feda8d49139edfaec138ccfa02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:13.181208 10901 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbaac6feda8d49139edfaec138ccfa02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbaac6feda8d49139edfaec138ccfa02, State: Initialized, Role: FOLLOWER
I20260924 03:21:13.181785 10901 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbaac6feda8d49139edfaec138ccfa02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbaac6feda8d49139edfaec138ccfa02" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36365 } }
I20260924 03:21:13.181998 10901 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbaac6feda8d49139edfaec138ccfa02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:21:13.182096 10901 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbaac6feda8d49139edfaec138ccfa02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:21:13.182228 10901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbaac6feda8d49139edfaec138ccfa02 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:13.184520 10901 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbaac6feda8d49139edfaec138ccfa02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbaac6feda8d49139edfaec138ccfa02" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36365 } }
I20260924 03:21:13.185117 10901 leader_election.cc:302] T 00000000000000000000000000000000 P bbaac6feda8d49139edfaec138ccfa02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbaac6feda8d49139edfaec138ccfa02; no voters:
I20260924 03:21:13.185843 10901 leader_election.cc:288] T 00000000000000000000000000000000 P bbaac6feda8d49139edfaec138ccfa02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:21:13.185982 10906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbaac6feda8d49139edfaec138ccfa02 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:13.186651 10906 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbaac6feda8d49139edfaec138ccfa02 [term 1 LEADER]: Becoming Leader. State: Replica: bbaac6feda8d49139edfaec138ccfa02, State: Running, Role: LEADER
I20260924 03:21:13.187234 10906 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbaac6feda8d49139edfaec138ccfa02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbaac6feda8d49139edfaec138ccfa02" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36365 } }
I20260924 03:21:13.188074 10901 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbaac6feda8d49139edfaec138ccfa02 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:21:13.190207 10907 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbaac6feda8d49139edfaec138ccfa02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbaac6feda8d49139edfaec138ccfa02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbaac6feda8d49139edfaec138ccfa02" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36365 } } }
I20260924 03:21:13.190675 10907 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbaac6feda8d49139edfaec138ccfa02 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:13.191164 10908 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbaac6feda8d49139edfaec138ccfa02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbaac6feda8d49139edfaec138ccfa02. Latest consensus state: current_term: 1 leader_uuid: "bbaac6feda8d49139edfaec138ccfa02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbaac6feda8d49139edfaec138ccfa02" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36365 } } }
I20260924 03:21:13.191483 10908 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbaac6feda8d49139edfaec138ccfa02 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:13.192411 10914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:21:13.198727 10914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:21:13.211365 10914 catalog_manager.cc:1347] Generated new cluster ID: 15ac99e1f1b04682aed650d6365c861a
I20260924 03:21:13.211495 10914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:21:13.225540 10914 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:21:13.227615 10914 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:21:13.244416 10914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbaac6feda8d49139edfaec138ccfa02: Generated new TSK 0
I20260924 03:21:13.245625 10914 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
I20260924 03:21:13.299230 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:36365
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:44711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:13.648653 10927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:13.648973 10927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:13.649104 10927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:13.657378 10927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:13.657513 10927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:13.657675 10927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:21:13.668025 10927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:44711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:36365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:13.669471 10927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:13.670936 10927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:13.685313 10933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:13.685263 10935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:13.685389 10932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:13.686492 10927 server_base.cc:1034] running on GCE node
I20260924 03:21:13.687165 10927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:13.688355 10927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:13.689852 10927 hybrid_clock.cc:648] HybridClock initialized: now 1790220073689679 us; error 44 us; skew 500 ppm
I20260924 03:21:13.690510 10927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:13 dist-test-slave-00qs krb5kdc[10821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220073, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:13.694409 10927 init.cc:377] Logged in from keytab as kudu/127.10.12.129@KRBTEST.COM (short username kudu)
I20260924 03:21:13.696506 10927 webserver.cc:460] Webserver started at http://127.10.12.129:44067/ using document root <none> and password file <none>
I20260924 03:21:13.697335 10927 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:13.697461 10927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:13.697796 10927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:13.701395 10927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "b2dc48b4d4b94c2ca5e62e0b2657fc0c"
format_stamp: "Formatted at 2026-09-24 03:21:13 on dist-test-slave-00qs"
server_key: "2f60791e293e85495ac680e8ccd7314b"
server_key_iv: "97578028222089eed2e2f294da84a558"
server_key_version: "encryptionkey@0"
I20260924 03:21:13.702320 10927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "b2dc48b4d4b94c2ca5e62e0b2657fc0c"
format_stamp: "Formatted at 2026-09-24 03:21:13 on dist-test-slave-00qs"
server_key: "2f60791e293e85495ac680e8ccd7314b"
server_key_iv: "97578028222089eed2e2f294da84a558"
server_key_version: "encryptionkey@0"
I20260924 03:21:13.708338 10927 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 03:21:13.712342 10942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:13.714476 10927 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:21:13.714670 10927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "b2dc48b4d4b94c2ca5e62e0b2657fc0c"
format_stamp: "Formatted at 2026-09-24 03:21:13 on dist-test-slave-00qs"
server_key: "2f60791e293e85495ac680e8ccd7314b"
server_key_iv: "97578028222089eed2e2f294da84a558"
server_key_version: "encryptionkey@0"
I20260924 03:21:13.714895 10927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:13.730705 10927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:13.735728 10927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:13.736066 10927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:13.737110 10927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:21:13.738608 10927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:13.738688 10927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:13.738776 10927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:13.738822 10927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:13.773938 10927 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:34473
I20260924 03:21:13.773981 11054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:34473 every 8 connection(s)
I20260924 03:21:13.775328 10927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:21:13.778481 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 10927
I20260924 03:21:13.778702 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:21:13.779525 10290 external_mini_cluster.cc:1428] Setting key 054a53340314af6370ecaac2e6fd1b61
Sep 24 03:21:13 dist-test-slave-00qs krb5kdc[10821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220073, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.12.130@KRBTEST.COM; defaulting to no policy
I20260924 03:21:13.797261 11057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:13.778085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:59267 (local address 127.10.12.190:36365)
0924 03:21:13.778538 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:13.778543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:13.779765 (+ 1222us) server_negotiation.cc:407] Connection header received
0924 03:21:13.781631 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:13.781636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:13.781726 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:13.781816 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:13.783382 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:13.784157 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:13.785771 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:13.786162 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:13.786280 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:13.789964 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:13.789972 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:13.789976 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:13.790016 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:13.793151 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:13.794380 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:13.794489 (+ 109us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:13.794493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:13.794659 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:13.795213 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:13.795217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:13.795219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:13.795523 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:13.795667 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:13.796576 (+ 909us) server_negotiation.cc:299] Negotiation successful
0924 03:21:13.796833 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":232,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.130@KRBTEST.COM" created.
I20260924 03:21:13.798301 11058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:13.778730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36365 (local address 127.10.12.129:59267)
0924 03:21:13.779567 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:13.779612 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:13.781366 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:13.782077 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:13.782087 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:13.782619 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:13.783207 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:13.783225 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:13.784324 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:13.784328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:13.785613 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:13.785627 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:13.786394 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:13.786401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:13.786634 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:13.787274 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:13.787298 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:13.789662 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:13.793388 (+ 3726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:13.793400 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:13.793421 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:13.794109 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:13.794834 (+ 725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:13.794840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:13.794844 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:13.795081 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:13.795662 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:13.795671 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:13.796114 (+ 443us) client_negotiation.cc:769] Sending connection context
0924 03:21:13.796439 (+ 325us) client_negotiation.cc:240] Negotiation successful
0924 03:21:13.797012 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":366,"threads_started":1}
I20260924 03:21:13.800149 11055 heartbeater.cc:347] Connected to a master server at 127.10.12.190:36365
I20260924 03:21:13.800462 11055 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:13.801218 11055 heartbeater.cc:511] Master 127.10.12.190:36365 requested a full tablet report, sending...
I20260924 03:21:13.803409 10866 ts_manager.cc:194] Registered new tserver with Master: b2dc48b4d4b94c2ca5e62e0b2657fc0c (127.10.12.129:34473)
I20260924 03:21:13.804970 10866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.129@KRBTEST.COM'} at 127.10.12.129:59267
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
I20260924 03:21:13.819690 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:36365
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:44711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:14.163111 11061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:14.163406 11061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:14.163508 11061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:14.171068 11061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:14.171205 11061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:14.171324 11061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:21:14.181555 11061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:44711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:36365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:14.182914 11061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:14.184347 11061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:14.199047 11066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:14.199103 11067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:14.199083 11069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:14.199755 11061 server_base.cc:1034] running on GCE node
I20260924 03:21:14.200559 11061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:14.201681 11061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:14.202926 11061 hybrid_clock.cc:648] HybridClock initialized: now 1790220074202880 us; error 35 us; skew 500 ppm
I20260924 03:21:14.203284 11061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:14 dist-test-slave-00qs krb5kdc[10821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220074, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:14.206930 11061 init.cc:377] Logged in from keytab as kudu/127.10.12.130@KRBTEST.COM (short username kudu)
I20260924 03:21:14.208741 11061 webserver.cc:460] Webserver started at http://127.10.12.130:44307/ using document root <none> and password file <none>
I20260924 03:21:14.209507 11061 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:14.209654 11061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:14.209986 11061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:14.213395 11061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "776b98d24a54473caaacc1fdfdf4fd37"
format_stamp: "Formatted at 2026-09-24 03:21:14 on dist-test-slave-00qs"
server_key: "1c29295dafa2c0fd3ce3df35836d1925"
server_key_iv: "b8b8874dc73c4d67548d7a462d9d1c11"
server_key_version: "encryptionkey@0"
I20260924 03:21:14.214293 11061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "776b98d24a54473caaacc1fdfdf4fd37"
format_stamp: "Formatted at 2026-09-24 03:21:14 on dist-test-slave-00qs"
server_key: "1c29295dafa2c0fd3ce3df35836d1925"
server_key_iv: "b8b8874dc73c4d67548d7a462d9d1c11"
server_key_version: "encryptionkey@0"
I20260924 03:21:14.220948 11061 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 03:21:14.225266 11076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:14.226951 11061 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 03:21:14.227134 11061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "776b98d24a54473caaacc1fdfdf4fd37"
format_stamp: "Formatted at 2026-09-24 03:21:14 on dist-test-slave-00qs"
server_key: "1c29295dafa2c0fd3ce3df35836d1925"
server_key_iv: "b8b8874dc73c4d67548d7a462d9d1c11"
server_key_version: "encryptionkey@0"
I20260924 03:21:14.227365 11061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:14.243156 11061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:14.248339 11061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:14.248651 11061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:14.249784 11061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:21:14.251327 11061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:14.251415 11061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:14.251519 11061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:14.251580 11061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:14.289019 11061 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:46459
I20260924 03:21:14.289088 11188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:46459 every 8 connection(s)
I20260924 03:21:14.290467 11061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:21:14.298848 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 11061
I20260924 03:21:14.299081 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:21:14.299522 10290 external_mini_cluster.cc:1428] Setting key 360303778588ead716c9f51fa947330f
Sep 24 03:21:14 dist-test-slave-00qs krb5kdc[10821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220074, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:14.315045 11057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:14.294454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:40907 (local address 127.10.12.190:36365)
0924 03:21:14.294696 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:14.294702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:14.296055 (+ 1353us) server_negotiation.cc:407] Connection header received
0924 03:21:14.298297 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:14.298303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:14.298401 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:14.298489 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:14.300469 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:14.301277 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:14.303220 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:14.303881 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:14.304069 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:14.308295 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:14.308302 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:14.308307 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:14.308348 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:14.311054 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:14.312256 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:14.312261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:14.312264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:14.312425 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:14.313060 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:14.313065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:14.313067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:14.313370 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:14.313505 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:14.314541 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0924 03:21:14.314750 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
WARNING: no policy specified for kudu/127.10.12.131@KRBTEST.COM; defaulting to no policy
I20260924 03:21:14.316569 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:14.294893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36365 (local address 127.10.12.130:40907)
0924 03:21:14.295842 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:14.295889 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:14.297956 (+ 2067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:14.298806 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:14.298816 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:14.299479 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:14.300240 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:14.300270 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:14.301472 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:14.301477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:14.303009 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:14.303026 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:14.304182 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:14.304191 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:14.304431 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:14.305287 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:14.305317 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:14.307903 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:14.311276 (+ 3373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:14.311289 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:14.311312 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:14.312077 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:14.312554 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:14.312559 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:14.312563 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:14.312899 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:14.313514 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:14.313523 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:14.314067 (+ 544us) client_negotiation.cc:769] Sending connection context
0924 03:21:14.314418 (+ 351us) client_negotiation.cc:240] Negotiation successful
0924 03:21:14.315070 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":348,"threads_started":1}
I20260924 03:21:14.318296 11189 heartbeater.cc:347] Connected to a master server at 127.10.12.190:36365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.131@KRBTEST.COM" created.
I20260924 03:21:14.318640 11189 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:14.319458 11189 heartbeater.cc:511] Master 127.10.12.190:36365 requested a full tablet report, sending...
I20260924 03:21:14.321280 10866 ts_manager.cc:194] Registered new tserver with Master: 776b98d24a54473caaacc1fdfdf4fd37 (127.10.12.130:46459)
I20260924 03:21:14.322679 10866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.130@KRBTEST.COM'} at 127.10.12.130:40907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
I20260924 03:21:14.340543 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:36365
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:44711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:14.678757 11194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:14.679080 11194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:14.679178 11194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:14.686820 11194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:14.686947 11194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:14.687067 11194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:21:14.697887 11194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:44711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:36365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:14.699211 11194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:14.700610 11194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:14.714435 11199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:14.714423 11202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:14.714468 11200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:14.715399 11194 server_base.cc:1034] running on GCE node
I20260924 03:21:14.716113 11194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:14.717216 11194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:14.718519 11194 hybrid_clock.cc:648] HybridClock initialized: now 1790220074718454 us; error 56 us; skew 500 ppm
I20260924 03:21:14.718955 11194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:14 dist-test-slave-00qs krb5kdc[10821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220074, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:14.722254 11194 init.cc:377] Logged in from keytab as kudu/127.10.12.131@KRBTEST.COM (short username kudu)
I20260924 03:21:14.723907 11194 webserver.cc:460] Webserver started at http://127.10.12.131:45023/ using document root <none> and password file <none>
I20260924 03:21:14.724679 11194 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:14.724814 11194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:14.725126 11194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:14.728354 11194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "a8e186dafe9249b4850ce0b15b6c531f"
format_stamp: "Formatted at 2026-09-24 03:21:14 on dist-test-slave-00qs"
server_key: "b77fc42ceb452f88b46c7834c234a047"
server_key_iv: "9bbbcd190bbfb6593857d9dae2c35e78"
server_key_version: "encryptionkey@0"
I20260924 03:21:14.729190 11194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "a8e186dafe9249b4850ce0b15b6c531f"
format_stamp: "Formatted at 2026-09-24 03:21:14 on dist-test-slave-00qs"
server_key: "b77fc42ceb452f88b46c7834c234a047"
server_key_iv: "9bbbcd190bbfb6593857d9dae2c35e78"
server_key_version: "encryptionkey@0"
I20260924 03:21:14.735281 11194 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 03:21:14.740103 11209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:14.742434 11194 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260924 03:21:14.742655 11194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "a8e186dafe9249b4850ce0b15b6c531f"
format_stamp: "Formatted at 2026-09-24 03:21:14 on dist-test-slave-00qs"
server_key: "b77fc42ceb452f88b46c7834c234a047"
server_key_iv: "9bbbcd190bbfb6593857d9dae2c35e78"
server_key_version: "encryptionkey@0"
I20260924 03:21:14.742879 11194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:14.792383 11194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:14.796922 11194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:14.797194 11194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:14.798298 11194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:21:14.799860 11194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:14.799984 11194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:14.800120 11194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:14.800168 11194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:14.808172 11055 heartbeater.cc:503] Master 127.10.12.190:36365 was elected leader, sending a full tablet report...
I20260924 03:21:14.831357 11194 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:39915
I20260924 03:21:14.831385 11321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:39915 every 8 connection(s)
I20260924 03:21:14.832928 11194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:21:14.839723 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 11194
I20260924 03:21:14.839921 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:21:14.840381 10290 external_mini_cluster.cc:1428] Setting key 9d55ee06c16f05a29e46521ee81e8a6d
Sep 24 03:21:14 dist-test-slave-00qs krb5kdc[10821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220074, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:14.854472 11324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:14.835659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:58279 (local address 127.10.12.190:36365)
0924 03:21:14.836121 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:14.836127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:14.837265 (+ 1138us) server_negotiation.cc:407] Connection header received
0924 03:21:14.839280 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:14.839287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:14.839395 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:14.839529 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:14.841551 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:14.842338 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:14.844066 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:14.844499 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:14.844614 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:14.848449 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:14.848454 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:14.848457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:14.848495 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:14.851118 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:14.851994 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:14.852000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:14.852005 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:14.852149 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:14.852601 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:14.852606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:14.852609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:14.852903 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:14.853048 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:14.853826 (+ 778us) server_negotiation.cc:299] Negotiation successful
0924 03:21:14.854078 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":210,"threads_started":1}
I20260924 03:21:14.855571 11325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:14.836335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36365 (local address 127.10.12.131:58279)
0924 03:21:14.837083 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:14.837130 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:14.838893 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:14.840087 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:14.840102 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:14.840745 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:14.841383 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:14.841403 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:14.842513 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:14.842517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:14.843877 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:14.843892 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:14.844742 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:14.844750 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:14.845026 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:14.845709 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:14.845745 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:14.848178 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:14.851318 (+ 3140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:14.851329 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:14.851347 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:14.851816 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:14.852262 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:14.852268 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:14.852272 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:14.852478 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:14.853041 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:14.853050 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:14.853431 (+ 381us) client_negotiation.cc:769] Sending connection context
0924 03:21:14.853689 (+ 258us) client_negotiation.cc:240] Negotiation successful
0924 03:21:14.854293 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":316,"threads_started":1}
I20260924 03:21:14.856940 11322 heartbeater.cc:347] Connected to a master server at 127.10.12.190:36365
I20260924 03:21:14.857308 11322 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:14.858114 11322 heartbeater.cc:511] Master 127.10.12.190:36365 requested a full tablet report, sending...
I20260924 03:21:14.859714 10866 ts_manager.cc:194] Registered new tserver with Master: a8e186dafe9249b4850ce0b15b6c531f (127.10.12.131:39915)
I20260924 03:21:14.860605 10866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.131@KRBTEST.COM'} at 127.10.12.131:58279
I20260924 03:21:14.870232 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:21:14.891933 11324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:14.877305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40052 (local address 127.10.12.190:36365)
0924 03:21:14.877536 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:14.877543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:14.877967 (+ 424us) server_negotiation.cc:407] Connection header received
0924 03:21:14.878160 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:14.878164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:14.878251 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:14.878370 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:14.880109 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:14.881004 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:14.882591 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:14.883195 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:14.883317 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:14.886107 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:14.886114 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:14.886118 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:14.886166 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:14.888964 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:14.889806 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:14.889810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:14.889813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:14.889961 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:14.890393 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:14.890397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:14.890399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:14.890781 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:14.890977 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:14.891362 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 03:21:14.891556 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 03:21:14.899331 10866 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:21:14.906987 10866 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:21:14.934654 11338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:14.922355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.10.12.131:39915)
0924 03:21:14.924591 (+ 2236us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:14.924600 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:14.924633 (+ 33us) server_negotiation.cc:407] Connection header received
0924 03:21:14.924738 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:14.924746 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:14.925054 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:14.925278 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:14.926358 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:14.928337 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:14.931696 (+ 3359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:14.933172 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:14.933292 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:14.933569 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:14.934076 (+ 507us) server_negotiation.cc:299] Negotiation successful
0924 03:21:14.934279 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2075,"thread_start_us":1915,"threads_started":1}
I20260924 03:21:14.934688 11339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:14.921866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:46459 (local address 127.0.0.1:51786)
0924 03:21:14.924077 (+ 2211us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:14.924101 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:14.924759 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:14.926320 (+ 1561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:14.926325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:14.926352 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:14.926612 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:14.926619 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:14.928718 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:14.928723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:14.931494 (+ 2771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:14.931531 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:14.933528 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:14.933535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:14.933896 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:14.933930 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 03:21:14.934022 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 03:21:14.934287 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2010,"mutex_wait_us":268,"thread_start_us":3986,"threads_started":1}
I20260924 03:21:14.934684 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:14.921889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51786 (local address 127.10.12.130:46459)
0924 03:21:14.923506 (+ 1617us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:14.923515 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:14.925653 (+ 2138us) server_negotiation.cc:407] Connection header received
0924 03:21:14.925836 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:14.925844 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:14.926098 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:14.926781 (+ 683us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:14.927024 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:14.928562 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:14.931678 (+ 3116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:14.933243 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:14.933406 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:14.933600 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:14.934176 (+ 576us) server_negotiation.cc:299] Negotiation successful
0924 03:21:14.934360 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1474,"thread_start_us":1341,"threads_started":1}
I20260924 03:21:14.935714 11335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:14.921585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:39915 (local address 127.0.0.1:42672)
0924 03:21:14.922676 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:14.922723 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:14.922928 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:14.925486 (+ 2558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:14.925494 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:14.925545 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:14.926149 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:14.926163 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:14.928482 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:14.928488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:14.931464 (+ 2976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:14.931486 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:14.933470 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:14.933478 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:14.933782 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:14.933867 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 03:21:14.933991 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 03:21:14.935300 (+ 1309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"mutex_wait_us":195,"thread_start_us":628,"threads_started":1}
I20260924 03:21:14.936666 11336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:14.923398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:34473 (local address 127.0.0.1:57922)
0924 03:21:14.926110 (+ 2712us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:14.926127 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:14.926243 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:14.926862 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:14.926867 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:14.926891 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:14.927240 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:14.927251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:14.929054 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:14.929058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:14.933748 (+ 4690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:14.933764 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:14.935692 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:14.935702 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:14.936025 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:14.936058 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 03:21:14.936156 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 03:21:14.936286 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2591,"mutex_wait_us":133,"thread_start_us":309,"threads_started":1}
I20260924 03:21:14.937145 11340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:14.924030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57922 (local address 127.10.12.129:34473)
0924 03:21:14.925381 (+ 1351us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:14.925389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:14.926228 (+ 839us) server_negotiation.cc:407] Connection header received
0924 03:21:14.926435 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:14.926443 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:14.926667 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:14.926854 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:14.927461 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:14.928907 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:14.933959 (+ 5052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:14.935459 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:14.935602 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:14.935826 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:14.936216 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 03:21:14.936397 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1103,"thread_start_us":961,"threads_started":1}
I20260924 03:21:14.939523 11124 tablet_service.cc:1467] Processing CreateTablet for tablet 313912206d344efc9c51c54bb8d5979f (DEFAULT_TABLE table=test-table [id=979916dea8dc45b98fc2cbfff9e9b37a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:14.940265 10990 tablet_service.cc:1467] Processing CreateTablet for tablet 313912206d344efc9c51c54bb8d5979f (DEFAULT_TABLE table=test-table [id=979916dea8dc45b98fc2cbfff9e9b37a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:14.940856 11124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 313912206d344efc9c51c54bb8d5979f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:14.941777 10990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 313912206d344efc9c51c54bb8d5979f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:14.942871 11257 tablet_service.cc:1467] Processing CreateTablet for tablet 313912206d344efc9c51c54bb8d5979f (DEFAULT_TABLE table=test-table [id=979916dea8dc45b98fc2cbfff9e9b37a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:14.944422 11257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 313912206d344efc9c51c54bb8d5979f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:14.953527 11341 tablet_bootstrap.cc:492] T 313912206d344efc9c51c54bb8d5979f P 776b98d24a54473caaacc1fdfdf4fd37: Bootstrap starting.
I20260924 03:21:14.956887 11341 tablet_bootstrap.cc:654] T 313912206d344efc9c51c54bb8d5979f P 776b98d24a54473caaacc1fdfdf4fd37: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:14.957096 11342 tablet_bootstrap.cc:492] T 313912206d344efc9c51c54bb8d5979f P b2dc48b4d4b94c2ca5e62e0b2657fc0c: Bootstrap starting.
I20260924 03:21:14.959043 11341 log.cc:824] T 313912206d344efc9c51c54bb8d5979f P 776b98d24a54473caaacc1fdfdf4fd37: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:14.959571 11343 tablet_bootstrap.cc:492] T 313912206d344efc9c51c54bb8d5979f P a8e186dafe9249b4850ce0b15b6c531f: Bootstrap starting.
I20260924 03:21:14.960974 11342 tablet_bootstrap.cc:654] T 313912206d344efc9c51c54bb8d5979f P b2dc48b4d4b94c2ca5e62e0b2657fc0c: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:14.962702 11342 log.cc:824] T 313912206d344efc9c51c54bb8d5979f P b2dc48b4d4b94c2ca5e62e0b2657fc0c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:14.963562 11343 tablet_bootstrap.cc:654] T 313912206d344efc9c51c54bb8d5979f P a8e186dafe9249b4850ce0b15b6c531f: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:14.963752 11341 tablet_bootstrap.cc:492] T 313912206d344efc9c51c54bb8d5979f P 776b98d24a54473caaacc1fdfdf4fd37: No bootstrap required, opened a new log
I20260924 03:21:14.964077 11341 ts_tablet_manager.cc:1397] T 313912206d344efc9c51c54bb8d5979f P 776b98d24a54473caaacc1fdfdf4fd37: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260924 03:21:14.964759 11343 log.cc:824] T 313912206d344efc9c51c54bb8d5979f P a8e186dafe9249b4850ce0b15b6c531f: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:14.966116 11342 tablet_bootstrap.cc:492] T 313912206d344efc9c51c54bb8d5979f P b2dc48b4d4b94c2ca5e62e0b2657fc0c: No bootstrap required, opened a new log
I20260924 03:21:14.966418 11342 ts_tablet_manager.cc:1397] T 313912206d344efc9c51c54bb8d5979f P b2dc48b4d4b94c2ca5e62e0b2657fc0c: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 03:21:14.967621 11343 tablet_bootstrap.cc:492] T 313912206d344efc9c51c54bb8d5979f P a8e186dafe9249b4850ce0b15b6c531f: No bootstrap required, opened a new log
I20260924 03:21:14.967984 11343 ts_tablet_manager.cc:1397] T 313912206d344efc9c51c54bb8d5979f P a8e186dafe9249b4850ce0b15b6c531f: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260924 03:21:14.975843 11341 raft_consensus.cc:348] T 313912206d344efc9c51c54bb8d5979f P 776b98d24a54473caaacc1fdfdf4fd37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8e186dafe9249b4850ce0b15b6c531f" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39915 } } peers { permanent_uuid: "776b98d24a54473caaacc1fdfdf4fd37" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 46459 } } peers { permanent_uuid: "b2dc48b4d4b94c2ca5e62e0b2657fc0c" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34473 } }
I20260924 03:21:14.976423 11341 raft_consensus.cc:374] T 313912206d344efc9c51c54bb8d5979f P 776b98d24a54473caaacc1fdfdf4fd37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:14.976194 11343 raft_consensus.cc:348] T 313912206d344efc9c51c54bb8d5979f P a8e186dafe9249b4850ce0b15b6c531f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8e186dafe9249b4850ce0b15b6c531f" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39915 } } peers { permanent_uuid: "776b98d24a54473caaacc1fdfdf4fd37" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 46459 } } peers { permanent_uuid: "b2dc48b4d4b94c2ca5e62e0b2657fc0c" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34473 } }
I20260924 03:21:14.976564 11341 raft_consensus.cc:729] T 313912206d344efc9c51c54bb8d5979f P 776b98d24a54473caaacc1fdfdf4fd37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 776b98d24a54473caaacc1fdfdf4fd37, State: Initialized, Role: FOLLOWER
I20260924 03:21:14.976588 11343 raft_consensus.cc:374] T 313912206d344efc9c51c54bb8d5979f P a8e186dafe9249b4850ce0b15b6c531f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:14.976686 11343 raft_consensus.cc:729] T 313912206d344efc9c51c54bb8d5979f P a8e186dafe9249b4850ce0b15b6c531f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8e186dafe9249b4850ce0b15b6c531f, State: Initialized, Role: FOLLOWER
I20260924 03:21:14.977331 11343 consensus_queue.cc:260] T 313912206d344efc9c51c54bb8d5979f P a8e186dafe9249b4850ce0b15b6c531f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8e186dafe9249b4850ce0b15b6c531f" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39915 } } peers { permanent_uuid: "776b98d24a54473caaacc1fdfdf4fd37" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 46459 } } peers { permanent_uuid: "b2dc48b4d4b94c2ca5e62e0b2657fc0c" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34473 } }
I20260924 03:21:14.977308 11341 consensus_queue.cc:260] T 313912206d344efc9c51c54bb8d5979f P 776b98d24a54473caaacc1fdfdf4fd37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8e186dafe9249b4850ce0b15b6c531f" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39915 } } peers { permanent_uuid: "776b98d24a54473caaacc1fdfdf4fd37" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 46459 } } peers { permanent_uuid: "b2dc48b4d4b94c2ca5e62e0b2657fc0c" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34473 } }
I20260924 03:21:14.977586 11342 raft_consensus.cc:348] T 313912206d344efc9c51c54bb8d5979f P b2dc48b4d4b94c2ca5e62e0b2657fc0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8e186dafe9249b4850ce0b15b6c531f" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39915 } } peers { permanent_uuid: "776b98d24a54473caaacc1fdfdf4fd37" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 46459 } } peers { permanent_uuid: "b2dc48b4d4b94c2ca5e62e0b2657fc0c" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34473 } }
I20260924 03:21:14.978138 11342 raft_consensus.cc:374] T 313912206d344efc9c51c54bb8d5979f P b2dc48b4d4b94c2ca5e62e0b2657fc0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:14.978288 11342 raft_consensus.cc:729] T 313912206d344efc9c51c54bb8d5979f P b2dc48b4d4b94c2ca5e62e0b2657fc0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2dc48b4d4b94c2ca5e62e0b2657fc0c, State: Initialized, Role: FOLLOWER
I20260924 03:21:14.979202 11342 consensus_queue.cc:260] T 313912206d344efc9c51c54bb8d5979f P b2dc48b4d4b94c2ca5e62e0b2657fc0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8e186dafe9249b4850ce0b15b6c531f" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39915 } } peers { permanent_uuid: "776b98d24a54473caaacc1fdfdf4fd37" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 46459 } } peers { permanent_uuid: "b2dc48b4d4b94c2ca5e62e0b2657fc0c" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34473 } }
I20260924 03:21:14.980489 11322 heartbeater.cc:503] Master 127.10.12.190:36365 was elected leader, sending a full tablet report...
I20260924 03:21:14.981324 11343 ts_tablet_manager.cc:1428] T 313912206d344efc9c51c54bb8d5979f P a8e186dafe9249b4850ce0b15b6c531f: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260924 03:21:14.983019 11189 heartbeater.cc:503] Master 127.10.12.190:36365 was elected leader, sending a full tablet report...
I20260924 03:21:14.983778 11341 ts_tablet_manager.cc:1428] T 313912206d344efc9c51c54bb8d5979f P 776b98d24a54473caaacc1fdfdf4fd37: Time spent starting tablet: real 0.020s user 0.013s sys 0.005s
I20260924 03:21:14.985294 11342 ts_tablet_manager.cc:1428] T 313912206d344efc9c51c54bb8d5979f P b2dc48b4d4b94c2ca5e62e0b2657fc0c: Time spent starting tablet: real 0.019s user 0.013s sys 0.001s
W20260924 03:21:15.028117 11056 tablet.cc:2369] T 313912206d344efc9c51c54bb8d5979f P b2dc48b4d4b94c2ca5e62e0b2657fc0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:21:15.042850 11190 tablet.cc:2369] T 313912206d344efc9c51c54bb8d5979f P 776b98d24a54473caaacc1fdfdf4fd37: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:21:15.084712 11323 tablet.cc:2369] T 313912206d344efc9c51c54bb8d5979f P a8e186dafe9249b4850ce0b15b6c531f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:21:15.106997 11349 raft_consensus.cc:482] T 313912206d344efc9c51c54bb8d5979f P b2dc48b4d4b94c2ca5e62e0b2657fc0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:21:15.107331 11349 raft_consensus.cc:504] T 313912206d344efc9c51c54bb8d5979f P b2dc48b4d4b94c2ca5e62e0b2657fc0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8e186dafe9249b4850ce0b15b6c531f" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39915 } } peers { permanent_uuid: "776b98d24a54473caaacc1fdfdf4fd37" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 46459 } } peers { permanent_uuid: "b2dc48b4d4b94c2ca5e62e0b2657fc0c" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34473 } }
I20260924 03:21:15.109737 11349 leader_election.cc:288] T 313912206d344efc9c51c54bb8d5979f P b2dc48b4d4b94c2ca5e62e0b2657fc0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8e186dafe9249b4850ce0b15b6c531f (127.10.12.131:39915), 776b98d24a54473caaacc1fdfdf4fd37 (127.10.12.130:46459)
I20260924 03:21:15.116012 11338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:15.109580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:43121 (local address 127.10.12.131:39915)
0924 03:21:15.109819 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:15.109825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:15.110869 (+ 1044us) server_negotiation.cc:407] Connection header received
0924 03:21:15.111026 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:15.111030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:15.111086 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:15.111160 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:15.111757 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:15.112463 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:15.114277 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:15.115061 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:15.115224 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:15.115306 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:15.115610 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 03:21:15.115745 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 03:21:15.116046 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:15.110005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:50479 (local address 127.10.12.130:46459)
0924 03:21:15.110256 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:15.110262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:15.110661 (+ 399us) server_negotiation.cc:407] Connection header received
0924 03:21:15.110869 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:15.110873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:15.110936 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:15.111011 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:15.111559 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:15.112317 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:15.114298 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:15.115074 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:15.115231 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:15.115307 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:15.115641 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 03:21:15.115769 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 03:21:15.116199 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:15.109544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:46459 (local address 127.10.12.129:50479)
0924 03:21:15.110514 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:15.110532 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:15.110724 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:15.111038 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:15.111043 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:15.111066 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:15.111392 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:15.111402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:15.112445 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:15.112451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:15.114137 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:15.114150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:15.115360 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:15.115366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:15.115494 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:15.115517 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 03:21:15.115595 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 03:21:15.115723 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":45,"thread_start_us":686,"threads_started":1}
I20260924 03:21:15.116245 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:15.109959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:39915 (local address 127.10.12.129:43121)
0924 03:21:15.110754 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:15.110769 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:15.110901 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:15.111212 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:15.111217 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:15.111243 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:15.111605 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:15.111613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:15.112606 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:15.112613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:15.114104 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:15.114126 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:15.115303 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:15.115309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:15.115458 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:15.115485 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 03:21:15.115567 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 03:21:15.115710 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":6,"thread_start_us":446,"threads_started":1}
I20260924 03:21:15.117265 11277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313912206d344efc9c51c54bb8d5979f" candidate_uuid: "b2dc48b4d4b94c2ca5e62e0b2657fc0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8e186dafe9249b4850ce0b15b6c531f" is_pre_election: true
I20260924 03:21:15.117273 11144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313912206d344efc9c51c54bb8d5979f" candidate_uuid: "b2dc48b4d4b94c2ca5e62e0b2657fc0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "776b98d24a54473caaacc1fdfdf4fd37" is_pre_election: true
I20260924 03:21:15.117730 11144 raft_consensus.cc:2454] T 313912206d344efc9c51c54bb8d5979f P 776b98d24a54473caaacc1fdfdf4fd37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2dc48b4d4b94c2ca5e62e0b2657fc0c in term 0.
I20260924 03:21:15.117730 11277 raft_consensus.cc:2454] T 313912206d344efc9c51c54bb8d5979f P a8e186dafe9249b4850ce0b15b6c531f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2dc48b4d4b94c2ca5e62e0b2657fc0c in term 0.
I20260924 03:21:15.118418 10944 leader_election.cc:302] T 313912206d344efc9c51c54bb8d5979f P b2dc48b4d4b94c2ca5e62e0b2657fc0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 776b98d24a54473caaacc1fdfdf4fd37, b2dc48b4d4b94c2ca5e62e0b2657fc0c; no voters:
I20260924 03:21:15.119372 11349 raft_consensus.cc:2781] T 313912206d344efc9c51c54bb8d5979f P b2dc48b4d4b94c2ca5e62e0b2657fc0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:21:15.119508 11349 raft_consensus.cc:482] T 313912206d344efc9c51c54bb8d5979f P b2dc48b4d4b94c2ca5e62e0b2657fc0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:21:15.119633 11349 raft_consensus.cc:3037] T 313912206d344efc9c51c54bb8d5979f P b2dc48b4d4b94c2ca5e62e0b2657fc0c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:15.122282 11349 raft_consensus.cc:504] T 313912206d344efc9c51c54bb8d5979f P b2dc48b4d4b94c2ca5e62e0b2657fc0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8e186dafe9249b4850ce0b15b6c531f" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39915 } } peers { permanent_uuid: "776b98d24a54473caaacc1fdfdf4fd37" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 46459 } } peers { permanent_uuid: "b2dc48b4d4b94c2ca5e62e0b2657fc0c" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34473 } }
I20260924 03:21:15.122912 11349 leader_election.cc:288] T 313912206d344efc9c51c54bb8d5979f P b2dc48b4d4b94c2ca5e62e0b2657fc0c [CANDIDATE]: Term 1 election: Requested vote from peers a8e186dafe9249b4850ce0b15b6c531f (127.10.12.131:39915), 776b98d24a54473caaacc1fdfdf4fd37 (127.10.12.130:46459)
I20260924 03:21:15.123437 11144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313912206d344efc9c51c54bb8d5979f" candidate_uuid: "b2dc48b4d4b94c2ca5e62e0b2657fc0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "776b98d24a54473caaacc1fdfdf4fd37"
I20260924 03:21:15.123481 11277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313912206d344efc9c51c54bb8d5979f" candidate_uuid: "b2dc48b4d4b94c2ca5e62e0b2657fc0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8e186dafe9249b4850ce0b15b6c531f"
I20260924 03:21:15.123632 11144 raft_consensus.cc:3037] T 313912206d344efc9c51c54bb8d5979f P 776b98d24a54473caaacc1fdfdf4fd37 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:15.123684 11277 raft_consensus.cc:3037] T 313912206d344efc9c51c54bb8d5979f P a8e186dafe9249b4850ce0b15b6c531f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:15.125873 11144 raft_consensus.cc:2454] T 313912206d344efc9c51c54bb8d5979f P 776b98d24a54473caaacc1fdfdf4fd37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2dc48b4d4b94c2ca5e62e0b2657fc0c in term 1.
I20260924 03:21:15.125969 11277 raft_consensus.cc:2454] T 313912206d344efc9c51c54bb8d5979f P a8e186dafe9249b4850ce0b15b6c531f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2dc48b4d4b94c2ca5e62e0b2657fc0c in term 1.
I20260924 03:21:15.126430 10944 leader_election.cc:302] T 313912206d344efc9c51c54bb8d5979f P b2dc48b4d4b94c2ca5e62e0b2657fc0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 776b98d24a54473caaacc1fdfdf4fd37, b2dc48b4d4b94c2ca5e62e0b2657fc0c; no voters:
I20260924 03:21:15.126716 11349 raft_consensus.cc:2781] T 313912206d344efc9c51c54bb8d5979f P b2dc48b4d4b94c2ca5e62e0b2657fc0c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:15.127259 11349 raft_consensus.cc:686] T 313912206d344efc9c51c54bb8d5979f P b2dc48b4d4b94c2ca5e62e0b2657fc0c [term 1 LEADER]: Becoming Leader. State: Replica: b2dc48b4d4b94c2ca5e62e0b2657fc0c, State: Running, Role: LEADER
I20260924 03:21:15.127779 11349 consensus_queue.cc:237] T 313912206d344efc9c51c54bb8d5979f P b2dc48b4d4b94c2ca5e62e0b2657fc0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8e186dafe9249b4850ce0b15b6c531f" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39915 } } peers { permanent_uuid: "776b98d24a54473caaacc1fdfdf4fd37" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 46459 } } peers { permanent_uuid: "b2dc48b4d4b94c2ca5e62e0b2657fc0c" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34473 } }
I20260924 03:21:15.132436 10864 catalog_manager.cc:5495] T 313912206d344efc9c51c54bb8d5979f P b2dc48b4d4b94c2ca5e62e0b2657fc0c reported cstate change: term changed from 0 to 1, leader changed from <none> to b2dc48b4d4b94c2ca5e62e0b2657fc0c (127.10.12.129). New cstate: current_term: 1 leader_uuid: "b2dc48b4d4b94c2ca5e62e0b2657fc0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8e186dafe9249b4850ce0b15b6c531f" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "776b98d24a54473caaacc1fdfdf4fd37" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 46459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2dc48b4d4b94c2ca5e62e0b2657fc0c" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34473 } health_report { overall_health: HEALTHY } } }
Sep 24 03:21:15 dist-test-slave-00qs krb5kdc[10821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.129@KRBTEST.COM
I20260924 03:21:15.189013 11340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:15.171026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57938 (local address 127.10.12.129:34473)
0924 03:21:15.171300 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:15.171306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:15.171404 (+ 98us) server_negotiation.cc:407] Connection header received
0924 03:21:15.171619 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:15.171623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:15.171708 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:15.171800 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:15.173425 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:15.174136 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:15.175421 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:15.175824 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:15.175944 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:15.181225 (+ 5281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:15.181244 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:15.181261 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:15.181307 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:15.185123 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:15.186140 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:15.186151 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:15.186154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:15.186418 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:15.186977 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:15.186984 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:15.186987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:15.187584 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:15.187929 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:15.188464 (+ 535us) server_negotiation.cc:299] Negotiation successful
0924 03:21:15.188746 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
Sep 24 03:21:15 dist-test-slave-00qs krb5kdc[10821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220075, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 03:21:15 dist-test-slave-00qs krb5kdc[10821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220075, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.12.129@KRBTEST.COM
I20260924 03:21:15.240041 11340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:15.223673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57948 (local address 127.10.12.129:34473)
0924 03:21:15.224005 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:15.224012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:15.224239 (+ 227us) server_negotiation.cc:407] Connection header received
0924 03:21:15.224441 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:15.224447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:15.224536 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:15.224658 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:15.226834 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:15.227592 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:15.229139 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:15.229595 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:15.229708 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:15.234435 (+ 4727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:15.234441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:15.234444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:15.234489 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:15.237256 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:15.238034 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:15.238038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:15.238041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:15.238182 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:15.238590 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:15.238594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:15.238596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:15.238911 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:15.239193 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:15.239441 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 03:21:15.239678 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
Sep 24 03:21:15 dist-test-slave-00qs krb5kdc[10821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 03:21:15 dist-test-slave-00qs krb5kdc[10821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.129@KRBTEST.COM
I20260924 03:21:15.287461 11340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:15.270804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57956 (local address 127.10.12.129:34473)
0924 03:21:15.271037 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:15.271043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:15.271392 (+ 349us) server_negotiation.cc:407] Connection header received
0924 03:21:15.271566 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:15.271571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:15.271656 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:15.271746 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:15.273450 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:15.274593 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:15.276173 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:15.276656 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:15.276820 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:15.281310 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:15.281319 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:15.281323 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:15.281384 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:15.284622 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:15.285509 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:15.285514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:15.285517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:15.285658 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:15.286061 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:15.286065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:15.286068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:15.286381 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:15.286578 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:15.286928 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 03:21:15.287119 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260924 03:21:15.289444 10970 tablet_service.cc:2908] Rejecting scan request for tablet 313912206d344efc9c51c54bb8d5979f: Uninitialized: clean time has not yet been initialized
I20260924 03:21:15.293242 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 10927
I20260924 03:21:15.308912 11340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:15.294476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57962 (local address 127.10.12.129:34473)
0924 03:21:15.294687 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:15.294693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:15.295051 (+ 358us) server_negotiation.cc:407] Connection header received
0924 03:21:15.295162 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:15.295166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:15.295245 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:15.295338 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:15.296958 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:15.297873 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:15.299696 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:15.300221 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:15.300336 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:15.303235 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:15.303240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:15.303244 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:15.303283 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:15.306173 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:15.306943 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:15.306948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:15.306950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:15.307077 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:15.307419 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:15.307423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:15.307425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:15.307715 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:15.307916 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:15.308328 (+ 412us) server_negotiation.cc:299] Negotiation successful
0924 03:21:15.308513 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 03:21:15.309579 11050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:15.402926 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 10927
I20260924 03:21:15.416937 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 11061
Sep 24 03:21:15 dist-test-slave-00qs krb5kdc[10821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.130@KRBTEST.COM
I20260924 03:21:15.432821 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:15.417925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51794 (local address 127.10.12.130:46459)
0924 03:21:15.418114 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:15.418119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:15.418138 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:21:15.418196 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:15.418200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:15.418262 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:15.418361 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:15.419524 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:15.420414 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:15.421775 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:15.422234 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:15.422347 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:15.426483 (+ 4136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:15.426502 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:15.426519 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:15.426564 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:15.429492 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:15.430351 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:15.430360 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:15.430362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:15.430610 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:15.431133 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:15.431139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:15.431142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:15.431560 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:15.431886 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:15.432279 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 03:21:15.432471 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 03:21:15.433487 11184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:15.522075 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 11061
I20260924 03:21:15.535936 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 11194
Sep 24 03:21:15 dist-test-slave-00qs krb5kdc[10821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.131@KRBTEST.COM
I20260924 03:21:15.551739 11338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:15.536757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42676 (local address 127.10.12.131:39915)
0924 03:21:15.536944 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:15.536949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:15.536969 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:21:15.537028 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:15.537032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:15.537094 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:15.537201 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:15.538455 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:15.539229 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:15.540777 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:15.541235 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:15.541345 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:15.545525 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:15.545543 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:15.545559 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:15.545601 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:15.548563 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:15.549652 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:15.549660 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:15.549662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:15.549849 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:15.550255 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:15.550259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:15.550261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:15.550657 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:15.550927 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:15.551262 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 03:21:15.551438 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 03:21:15.552421 11317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:15.641608 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 11194
I20260924 03:21:15.654619 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 10834
I20260924 03:21:15.655860 10896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:15.766812 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 10834
2026-09-24T03:21:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4290 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:21:15 dist-test-slave-00qs krb5kdc[11371](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:21:15 dist-test-slave-00qs krb5kdc[11371](info): set up 2 sockets
Sep 24 03:21:15 dist-test-slave-00qs krb5kdc[11371](info): commencing operation
krb5kdc: starting...
W20260924 03:21:16.784147 10290 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:21:16 dist-test-slave-00qs krb5kdc[11371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:21:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:21:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:21:16.973203 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:43383
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:45829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:43383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:17.307626 11384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:17.308017 11384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:17.308161 11384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:17.315713 11384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:21:17.315871 11384 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 03:21:17.315935 11384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:17.316044 11384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:17.316111 11384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:21:17.316195 11384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:21:17.326469 11384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:45829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:43383
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:43383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:17.327863 11384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:17.329224 11384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:17.341038 11389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:17.340993 11392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:17.341118 11390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:17.342325 11384 server_base.cc:1034] running on GCE node
I20260924 03:21:17.343215 11384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:17.344899 11384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:17.346172 11384 hybrid_clock.cc:648] HybridClock initialized: now 1790220077346085 us; error 57 us; skew 500 ppm
I20260924 03:21:17.346566 11384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:17 dist-test-slave-00qs krb5kdc[11371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220077, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:17.350334 11384 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:21:17.352352 11384 webserver.cc:460] Webserver started at http://127.10.12.190:42345/ using document root <none> and password file <none>
I20260924 03:21:17.353137 11384 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:17.353251 11384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:17.353575 11384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:17.356717 11384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "add047dafdab4e0fbbedef99257fca6b"
format_stamp: "Formatted at 2026-09-24 03:21:17 on dist-test-slave-00qs"
server_key: "7c46deb9c8eba32b4f6afd3beb33491e"
server_key_iv: "7a3d624ee2f11194edba2a5b37bcee6d"
server_key_version: "encryptionkey@0"
I20260924 03:21:17.357515 11384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "add047dafdab4e0fbbedef99257fca6b"
format_stamp: "Formatted at 2026-09-24 03:21:17 on dist-test-slave-00qs"
server_key: "7c46deb9c8eba32b4f6afd3beb33491e"
server_key_iv: "7a3d624ee2f11194edba2a5b37bcee6d"
server_key_version: "encryptionkey@0"
I20260924 03:21:17.363466 11384 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 03:21:17.367921 11399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:17.369551 11384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:21:17.369762 11384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "add047dafdab4e0fbbedef99257fca6b"
format_stamp: "Formatted at 2026-09-24 03:21:17 on dist-test-slave-00qs"
server_key: "7c46deb9c8eba32b4f6afd3beb33491e"
server_key_iv: "7a3d624ee2f11194edba2a5b37bcee6d"
server_key_version: "encryptionkey@0"
I20260924 03:21:17.370014 11384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:17.392261 11384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:17.396819 11384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:17.397111 11384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:17.419302 11384 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:43383
I20260924 03:21:17.419406 11460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:43383 every 8 connection(s)
I20260924 03:21:17.421625 11384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:21:17.422466 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 11384
I20260924 03:21:17.422686 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:21:17.423487 10290 external_mini_cluster.cc:1428] Setting key 566cf493e2c189016540d711c1196334
I20260924 03:21:17.426833 11461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 03:21:17 dist-test-slave-00qs krb5kdc[11371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:17.439050 11461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P add047dafdab4e0fbbedef99257fca6b: Bootstrap starting.
I20260924 03:21:17.443068 11461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P add047dafdab4e0fbbedef99257fca6b: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:17.444346 11461 log.cc:824] T 00000000000000000000000000000000 P add047dafdab4e0fbbedef99257fca6b: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:17.447638 11461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P add047dafdab4e0fbbedef99257fca6b: No bootstrap required, opened a new log
I20260924 03:21:17.449155 11464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:17.425776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52754 (local address 127.10.12.190:43383)
0924 03:21:17.426635 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:17.426648 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:17.426689 (+ 41us) server_negotiation.cc:407] Connection header received
0924 03:21:17.428788 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:17.428816 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:17.429285 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:17.429736 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:17.431336 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:17.432938 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:17.434334 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:17.435123 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:17.435313 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:17.440806 (+ 5493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:17.440816 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:17.440836 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:17.440888 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:17.444645 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:17.445523 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:17.445532 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:17.445541 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:17.445743 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:17.446244 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:17.446248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:17.446251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:17.446831 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:17.447076 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:17.447610 (+ 534us) server_negotiation.cc:299] Negotiation successful
0924 03:21:17.448066 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":405,"threads_started":1}
I20260924 03:21:17.456883 11461 raft_consensus.cc:348] T 00000000000000000000000000000000 P add047dafdab4e0fbbedef99257fca6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add047dafdab4e0fbbedef99257fca6b" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 43383 } }
I20260924 03:21:17.457275 11461 raft_consensus.cc:374] T 00000000000000000000000000000000 P add047dafdab4e0fbbedef99257fca6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:17.457378 11461 raft_consensus.cc:729] T 00000000000000000000000000000000 P add047dafdab4e0fbbedef99257fca6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: add047dafdab4e0fbbedef99257fca6b, State: Initialized, Role: FOLLOWER
I20260924 03:21:17.458022 11461 consensus_queue.cc:260] T 00000000000000000000000000000000 P add047dafdab4e0fbbedef99257fca6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add047dafdab4e0fbbedef99257fca6b" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 43383 } }
I20260924 03:21:17.458246 11461 raft_consensus.cc:388] T 00000000000000000000000000000000 P add047dafdab4e0fbbedef99257fca6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:21:17.458352 11461 raft_consensus.cc:482] T 00000000000000000000000000000000 P add047dafdab4e0fbbedef99257fca6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:21:17.458529 11461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P add047dafdab4e0fbbedef99257fca6b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:17.461067 11461 raft_consensus.cc:504] T 00000000000000000000000000000000 P add047dafdab4e0fbbedef99257fca6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add047dafdab4e0fbbedef99257fca6b" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 43383 } }
I20260924 03:21:17.461634 11461 leader_election.cc:302] T 00000000000000000000000000000000 P add047dafdab4e0fbbedef99257fca6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: add047dafdab4e0fbbedef99257fca6b; no voters:
I20260924 03:21:17.462242 11461 leader_election.cc:288] T 00000000000000000000000000000000 P add047dafdab4e0fbbedef99257fca6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:21:17.462388 11466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P add047dafdab4e0fbbedef99257fca6b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:17.463061 11466 raft_consensus.cc:686] T 00000000000000000000000000000000 P add047dafdab4e0fbbedef99257fca6b [term 1 LEADER]: Becoming Leader. State: Replica: add047dafdab4e0fbbedef99257fca6b, State: Running, Role: LEADER
I20260924 03:21:17.463641 11466 consensus_queue.cc:237] T 00000000000000000000000000000000 P add047dafdab4e0fbbedef99257fca6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add047dafdab4e0fbbedef99257fca6b" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 43383 } }
I20260924 03:21:17.464408 11461 sys_catalog.cc:564] T 00000000000000000000000000000000 P add047dafdab4e0fbbedef99257fca6b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:21:17.466517 11467 sys_catalog.cc:455] T 00000000000000000000000000000000 P add047dafdab4e0fbbedef99257fca6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "add047dafdab4e0fbbedef99257fca6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add047dafdab4e0fbbedef99257fca6b" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 43383 } } }
I20260924 03:21:17.466849 11467 sys_catalog.cc:458] T 00000000000000000000000000000000 P add047dafdab4e0fbbedef99257fca6b [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:17.467715 11472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:21:17.467530 11468 sys_catalog.cc:455] T 00000000000000000000000000000000 P add047dafdab4e0fbbedef99257fca6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader add047dafdab4e0fbbedef99257fca6b. Latest consensus state: current_term: 1 leader_uuid: "add047dafdab4e0fbbedef99257fca6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add047dafdab4e0fbbedef99257fca6b" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 43383 } } }
I20260924 03:21:17.468895 11468 sys_catalog.cc:458] T 00000000000000000000000000000000 P add047dafdab4e0fbbedef99257fca6b [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:17.474822 11472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:21:17.485574 11472 catalog_manager.cc:1347] Generated new cluster ID: ff49320991e14eed94409153db8c06b4
I20260924 03:21:17.485741 11472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:21:17.496219 11472 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:21:17.497576 11472 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:21:17.503822 11472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P add047dafdab4e0fbbedef99257fca6b: Generated new TSK 0
I20260924 03:21:17.505281 11472 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
I20260924 03:21:17.561825 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:43383
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:45829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:17.897964 11487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:17.898301 11487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:17.898439 11487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:17.906486 11487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:17.906646 11487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:17.906792 11487 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:21:17.906906 11487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:21:17.917279 11487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:45829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.12.190:43383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:17.918684 11487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:17.920184 11487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:17.935863 11493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:17.935740 11492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:17.935829 11495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:17.936113 11487 server_base.cc:1034] running on GCE node
I20260924 03:21:17.937040 11487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:17.938122 11487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:17.939369 11487 hybrid_clock.cc:648] HybridClock initialized: now 1790220077939299 us; error 42 us; skew 500 ppm
I20260924 03:21:17.939709 11487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:17 dist-test-slave-00qs krb5kdc[11371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220077, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:17.943212 11487 init.cc:377] Logged in from keytab as kudu/127.10.12.129@KRBTEST.COM (short username kudu)
I20260924 03:21:17.944993 11487 webserver.cc:460] Webserver started at http://127.10.12.129:45805/ using document root <none> and password file <none>
I20260924 03:21:17.945762 11487 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:17.945878 11487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:17.946200 11487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:17.949453 11487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "1c5ef5580b16496bad3459d19814ccd7"
format_stamp: "Formatted at 2026-09-24 03:21:17 on dist-test-slave-00qs"
server_key: "e1844875ef158e303ca1d0b9d7a1ad5b"
server_key_iv: "e88d2474284ae69f3772ab376cb77a01"
server_key_version: "encryptionkey@0"
I20260924 03:21:17.950393 11487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "1c5ef5580b16496bad3459d19814ccd7"
format_stamp: "Formatted at 2026-09-24 03:21:17 on dist-test-slave-00qs"
server_key: "e1844875ef158e303ca1d0b9d7a1ad5b"
server_key_iv: "e88d2474284ae69f3772ab376cb77a01"
server_key_version: "encryptionkey@0"
I20260924 03:21:17.956789 11487 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 03:21:17.961084 11502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:17.963047 11487 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260924 03:21:17.963251 11487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "1c5ef5580b16496bad3459d19814ccd7"
format_stamp: "Formatted at 2026-09-24 03:21:17 on dist-test-slave-00qs"
server_key: "e1844875ef158e303ca1d0b9d7a1ad5b"
server_key_iv: "e88d2474284ae69f3772ab376cb77a01"
server_key_version: "encryptionkey@0"
I20260924 03:21:17.963508 11487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:17.986940 11487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:17.992414 11487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:17.992674 11487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:17.995813 11487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:17.995909 11487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:17.996058 11487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:17.996132 11487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:21:18 dist-test-slave-00qs krb5kdc[11371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220077, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:18.024813 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:17.996990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:57651 (local address 127.10.12.190:43383)
0924 03:21:17.997435 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:17.997440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:17.998866 (+ 1426us) server_negotiation.cc:407] Connection header received
0924 03:21:18.002178 (+ 3312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:18.002185 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:18.002339 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:18.002476 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:18.004910 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:18.005779 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:18.009124 (+ 3345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:18.009841 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:18.009973 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:18.016693 (+ 6720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:18.016702 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:18.016706 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:18.016757 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:18.020462 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:18.021839 (+ 1377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:18.021846 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:18.021851 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:18.022023 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:18.022669 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:18.022674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:18.022677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:18.022995 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:18.023165 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:18.024314 (+ 1149us) server_negotiation.cc:299] Negotiation successful
0924 03:21:18.024521 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":224,"threads_started":1}
I20260924 03:21:18.026667 11512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:17.997743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:43383 (local address 127.10.12.129:57651)
0924 03:21:17.998688 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:17.998730 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:18.001756 (+ 3026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:18.002833 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:18.002847 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:18.003536 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:18.004552 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:18.004591 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:18.006050 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:18.006056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:18.008820 (+ 2764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:18.008868 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:18.010443 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:18.010454 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:18.010892 (+ 438us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:18.011852 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:18.011892 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:18.016260 (+ 4368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:18.020708 (+ 4448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:18.020720 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:18.020743 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:18.021599 (+ 856us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:18.022213 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:18.022219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:18.022223 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:18.022516 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:18.023168 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:18.023182 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:18.023768 (+ 586us) client_negotiation.cc:769] Sending connection context
0924 03:21:18.024158 (+ 390us) client_negotiation.cc:240] Negotiation successful
0924 03:21:18.024870 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":366,"threads_started":1}
I20260924 03:21:18.040181 11487 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:44489
I20260924 03:21:18.040207 11622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:44489 every 8 connection(s)
I20260924 03:21:18.041858 11487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:21:18.044821 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 11487
I20260924 03:21:18.045045 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:21:18.045063 11623 heartbeater.cc:347] Connected to a master server at 127.10.12.190:43383
I20260924 03:21:18.045325 11623 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:18.045847 10290 external_mini_cluster.cc:1428] Setting key cbae625fc53fa41a168bfa93fd8b8771
I20260924 03:21:18.046227 11623 heartbeater.cc:511] Master 127.10.12.190:43383 requested a full tablet report, sending...
I20260924 03:21:18.048866 11415 ts_manager.cc:194] Registered new tserver with Master: 1c5ef5580b16496bad3459d19814ccd7 (127.10.12.129:44489)
I20260924 03:21:18.051012 11415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.129@KRBTEST.COM'} at 127.10.12.129:57651
I20260924 03:21:18.053977 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:18.033486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:45985 (local address 127.10.12.190:43383)
0924 03:21:18.033708 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:18.033714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:18.040692 (+ 6978us) server_negotiation.cc:407] Connection header received
0924 03:21:18.040826 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:18.040832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:18.040925 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:18.041048 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:18.042137 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:18.043245 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:18.045149 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:18.045826 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:18.046000 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:18.048177 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:18.048183 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:18.048186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:18.048218 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:18.051052 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:18.051806 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:18.051811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:18.051813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:18.052021 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:18.052491 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:18.052496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:18.052499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:18.052794 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:18.052933 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:18.053366 (+ 433us) server_negotiation.cc:299] Negotiation successful
0924 03:21:18.053615 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 03:21:18.054093 11603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:18.033510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:43383 (local address 127.10.12.129:45985)
0924 03:21:18.040538 (+ 7028us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:18.040562 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:18.040723 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:18.041112 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:18.041116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:18.041616 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:18.041944 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:18.041955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:18.043413 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:18.043417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:18.044863 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:18.044881 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:18.046087 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:18.046095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:18.046283 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:18.046948 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:18.046973 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:18.047945 (+ 972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:18.051230 (+ 3285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:18.051236 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:18.051241 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:18.051656 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:18.052159 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:18.052164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:18.052179 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:18.052347 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:18.052918 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:18.052923 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:18.053116 (+ 193us) client_negotiation.cc:769] Sending connection context
0924 03:21:18.053234 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 03:21:18.053626 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6862,"mutex_wait_us":48,"thread_start_us":6679,"threads_started":1}
WARNING: no policy specified for kudu/127.10.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
I20260924 03:21:18.088995 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:43383
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:45829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:18.428403 11627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:18.428714 11627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:18.428876 11627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:18.436944 11627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:18.437080 11627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:18.437175 11627 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:21:18.437300 11627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:21:18.448132 11627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:45829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.12.190:43383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:18.449563 11627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:18.451006 11627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:18.465258 11633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:18.465369 11635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:18.465252 11632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:18.466480 11627 server_base.cc:1034] running on GCE node
I20260924 03:21:18.467154 11627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:18.468276 11627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:18.469542 11627 hybrid_clock.cc:648] HybridClock initialized: now 1790220078469461 us; error 52 us; skew 500 ppm
I20260924 03:21:18.469928 11627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:18 dist-test-slave-00qs krb5kdc[11371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220078, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:18.480402 11627 init.cc:377] Logged in from keytab as kudu/127.10.12.130@KRBTEST.COM (short username kudu)
I20260924 03:21:18.482203 11627 webserver.cc:460] Webserver started at http://127.10.12.130:46119/ using document root <none> and password file <none>
I20260924 03:21:18.483002 11627 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:18.483124 11627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:18.483469 11627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:18.486696 11627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "fcd549f721904765b980cba6e5fd4046"
format_stamp: "Formatted at 2026-09-24 03:21:18 on dist-test-slave-00qs"
server_key: "b9aa1a7c2709256af051c56c52a5af7d"
server_key_iv: "4bf6558d624dcc6cbacb4c77f9fc2e88"
server_key_version: "encryptionkey@0"
I20260924 03:21:18.487572 11627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "fcd549f721904765b980cba6e5fd4046"
format_stamp: "Formatted at 2026-09-24 03:21:18 on dist-test-slave-00qs"
server_key: "b9aa1a7c2709256af051c56c52a5af7d"
server_key_iv: "4bf6558d624dcc6cbacb4c77f9fc2e88"
server_key_version: "encryptionkey@0"
I20260924 03:21:18.495121 11627 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260924 03:21:18.500519 11642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:18.502411 11627 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 03:21:18.502627 11627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "fcd549f721904765b980cba6e5fd4046"
format_stamp: "Formatted at 2026-09-24 03:21:18 on dist-test-slave-00qs"
server_key: "b9aa1a7c2709256af051c56c52a5af7d"
server_key_iv: "4bf6558d624dcc6cbacb4c77f9fc2e88"
server_key_version: "encryptionkey@0"
I20260924 03:21:18.503003 11627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:18.526161 11627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:18.531224 11627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:18.531503 11627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:18.534386 11627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:18.534488 11627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:18.534612 11627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:18.534678 11627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:21:18 dist-test-slave-00qs krb5kdc[11371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220078, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:18.559684 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:18.534976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:43121 (local address 127.10.12.190:43383)
0924 03:21:18.536119 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:18.536127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:18.536726 (+ 599us) server_negotiation.cc:407] Connection header received
0924 03:21:18.539195 (+ 2469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:18.539203 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:18.539311 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:18.539436 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:18.541803 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:18.542543 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:18.544529 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:18.545131 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:18.545262 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:18.551870 (+ 6608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:18.551879 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:18.551884 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:18.551940 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:18.555627 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:18.556877 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:18.556884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:18.556888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:18.557045 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:18.557521 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:18.557526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:18.557528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:18.557833 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:18.557988 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:18.558983 (+ 995us) server_negotiation.cc:299] Negotiation successful
0924 03:21:18.559273 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":991}
I20260924 03:21:18.560552 11651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:18.535632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:43383 (local address 127.10.12.130:43121)
0924 03:21:18.536530 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:18.536582 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:18.538791 (+ 2209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:18.539771 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:18.539784 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:18.540635 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:18.541591 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:18.541615 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:18.542715 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:18.542721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:18.544288 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:18.544312 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:18.545449 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:18.545457 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:18.545844 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:18.546759 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:18.546806 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:18.551456 (+ 4650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:18.555848 (+ 4392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:18.555861 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:18.555886 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:18.556711 (+ 825us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:18.557168 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:18.557172 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:18.557175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:18.557396 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:18.557968 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:18.557977 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:18.558456 (+ 479us) client_negotiation.cc:769] Sending connection context
0924 03:21:18.558787 (+ 331us) client_negotiation.cc:240] Negotiation successful
0924 03:21:18.559236 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":409,"threads_started":1}
I20260924 03:21:18.581681 11712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:18.566708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:43383 (local address 127.10.12.130:53305)
0924 03:21:18.567620 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:18.567644 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:18.567988 (+ 344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:18.568613 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:18.568618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:18.569102 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:18.569402 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:18.569412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:18.570483 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:18.570488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:18.571619 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:18.571635 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:18.572424 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:18.572431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:18.572602 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:18.573387 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:18.573406 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:18.574556 (+ 1150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:18.578117 (+ 3561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:18.578124 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:18.578129 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:18.578778 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:18.579532 (+ 754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:18.579536 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:18.579539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:18.579723 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:18.580699 (+ 976us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:18.580704 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:18.580908 (+ 204us) client_negotiation.cc:769] Sending connection context
0924 03:21:18.581033 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 03:21:18.581330 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":391,"threads_started":1}
I20260924 03:21:18.581782 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:18.566649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:53305 (local address 127.10.12.190:43383)
0924 03:21:18.566877 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:18.566881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:18.567768 (+ 887us) server_negotiation.cc:407] Connection header received
0924 03:21:18.568181 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:18.568188 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:18.568275 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:18.568346 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:18.569556 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:18.570339 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:18.571770 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:18.572228 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:18.572343 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:18.574768 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:18.574774 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:18.574777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:18.574812 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:18.577932 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:18.579101 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:18.579113 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:18.579117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:18.579369 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:18.579915 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:18.579921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:18.579924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:18.580546 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:18.580734 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:18.581175 (+ 441us) server_negotiation.cc:299] Negotiation successful
0924 03:21:18.581460 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260924 03:21:18.591194 11627 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:36641
I20260924 03:21:18.591210 11761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:36641 every 8 connection(s)
I20260924 03:21:18.592549 11627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:21:18.594805 11762 heartbeater.cc:347] Connected to a master server at 127.10.12.190:43383
I20260924 03:21:18.595046 11762 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:18.595716 11762 heartbeater.cc:511] Master 127.10.12.190:43383 requested a full tablet report, sending...
I20260924 03:21:18.597187 11415 ts_manager.cc:194] Registered new tserver with Master: fcd549f721904765b980cba6e5fd4046 (127.10.12.130:36641)
I20260924 03:21:18.598057 11415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.130@KRBTEST.COM'} at 127.10.12.130:43121
I20260924 03:21:18.599368 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 11627
I20260924 03:21:18.599537 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:21:18.600008 10290 external_mini_cluster.cc:1428] Setting key 938030560d230f40da7bef46788f8557
WARNING: no policy specified for kudu/127.10.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
I20260924 03:21:18.644685 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:43383
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:45829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:18.984920 11766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:18.985251 11766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:18.985375 11766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:18.992950 11766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:18.993074 11766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:18.993184 11766 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:21:18.993286 11766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:21:19.003722 11766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:45829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.12.190:43383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:19.005111 11766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:19.006469 11766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:19.020537 11771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:19.020529 11774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:19.020488 11772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:19.021317 11766 server_base.cc:1034] running on GCE node
I20260924 03:21:19.021986 11766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:19.023025 11766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:19.024260 11766 hybrid_clock.cc:648] HybridClock initialized: now 1790220079024186 us; error 49 us; skew 500 ppm
I20260924 03:21:19.024581 11766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:19 dist-test-slave-00qs krb5kdc[11371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220079, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:19.027832 11766 init.cc:377] Logged in from keytab as kudu/127.10.12.131@KRBTEST.COM (short username kudu)
I20260924 03:21:19.029524 11766 webserver.cc:460] Webserver started at http://127.10.12.131:33141/ using document root <none> and password file <none>
I20260924 03:21:19.030275 11766 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:19.030386 11766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:19.030716 11766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:19.033838 11766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "e6759c961f724ee1ab5dd4bc22b3411b"
format_stamp: "Formatted at 2026-09-24 03:21:19 on dist-test-slave-00qs"
server_key: "d5b572578ec156a5a468596dbff3d9b5"
server_key_iv: "cd214c0befbb637283fd97aaff9a36db"
server_key_version: "encryptionkey@0"
I20260924 03:21:19.034682 11766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "e6759c961f724ee1ab5dd4bc22b3411b"
format_stamp: "Formatted at 2026-09-24 03:21:19 on dist-test-slave-00qs"
server_key: "d5b572578ec156a5a468596dbff3d9b5"
server_key_iv: "cd214c0befbb637283fd97aaff9a36db"
server_key_version: "encryptionkey@0"
I20260924 03:21:19.040436 11766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 03:21:19.044302 11781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:19.046065 11766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:21:19.046244 11766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "e6759c961f724ee1ab5dd4bc22b3411b"
format_stamp: "Formatted at 2026-09-24 03:21:19 on dist-test-slave-00qs"
server_key: "d5b572578ec156a5a468596dbff3d9b5"
server_key_iv: "cd214c0befbb637283fd97aaff9a36db"
server_key_version: "encryptionkey@0"
I20260924 03:21:19.046471 11766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:19.055225 11623 heartbeater.cc:503] Master 127.10.12.190:43383 was elected leader, sending a full tablet report...
I20260924 03:21:19.067143 11766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:19.071897 11766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:19.072201 11766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:19.075067 11766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:19.075165 11766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:19.075294 11766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:19.075367 11766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:21:19 dist-test-slave-00qs krb5kdc[11371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220079, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:19.102411 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:19.075843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:44281 (local address 127.10.12.190:43383)
0924 03:21:19.076057 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:19.076063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:19.077609 (+ 1546us) server_negotiation.cc:407] Connection header received
0924 03:21:19.080601 (+ 2992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:19.080608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:19.080712 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:19.080830 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:19.083171 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.084428 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.086787 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.087427 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.087579 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:19.093230 (+ 5651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:19.093239 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:19.093244 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:19.093309 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:19.097399 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:19.098731 (+ 1332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:19.098738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:19.098741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:19.098943 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:19.099605 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:19.099610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:19.099614 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:19.100056 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:19.100235 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:19.101456 (+ 1221us) server_negotiation.cc:299] Negotiation successful
0924 03:21:19.101760 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 03:21:19.103914 11790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:19.076483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:43383 (local address 127.10.12.131:44281)
0924 03:21:19.077399 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:19.077456 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:19.080143 (+ 2687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:19.081193 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:19.081209 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:19.082266 (+ 1057us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:19.082939 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:19.082961 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.084645 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.084655 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:19.086578 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:19.086598 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.087753 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.087766 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:19.088137 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:19.089078 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:19.089122 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:19.092848 (+ 3726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:19.097659 (+ 4811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:19.097674 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:19.097698 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:19.098544 (+ 846us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:19.099093 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:19.099099 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:19.099102 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:19.099464 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:19.100229 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:19.100242 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:19.100876 (+ 634us) client_negotiation.cc:769] Sending connection context
0924 03:21:19.101289 (+ 413us) client_negotiation.cc:240] Negotiation successful
0924 03:21:19.102002 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":340,"threads_started":1}
I20260924 03:21:19.117115 11766 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:40969
I20260924 03:21:19.117128 11900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:40969 every 8 connection(s)
I20260924 03:21:19.118981 11766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:21:19.121898 11901 heartbeater.cc:347] Connected to a master server at 127.10.12.190:43383
I20260924 03:21:19.122198 11901 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:19.123126 11901 heartbeater.cc:511] Master 127.10.12.190:43383 requested a full tablet report, sending...
I20260924 03:21:19.124713 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 11766
I20260924 03:21:19.124930 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:21:19.125218 11415 ts_manager.cc:194] Registered new tserver with Master: e6759c961f724ee1ab5dd4bc22b3411b (127.10.12.131:40969)
I20260924 03:21:19.125384 10290 external_mini_cluster.cc:1428] Setting key ff9f587da4eb7c8f8e42734795d9f39f
I20260924 03:21:19.126426 11415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.131@KRBTEST.COM'} at 127.10.12.131:44281
I20260924 03:21:19.128672 11890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:19.112577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:43383 (local address 127.10.12.131:41721)
0924 03:21:19.113477 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:19.113504 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:19.113717 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:19.114339 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:19.114345 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:19.114968 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:19.115309 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:19.115322 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.116681 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.116687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:19.117912 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:19.117930 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.118823 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.118830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:19.118995 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:19.119653 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:19.119673 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:19.121003 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:19.125141 (+ 4138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:19.125146 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:19.125150 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:19.125684 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:19.126188 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:19.126193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:19.126197 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:19.126363 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:19.127172 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:19.127176 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:19.127381 (+ 205us) client_negotiation.cc:769] Sending connection context
0924 03:21:19.127537 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 03:21:19.127917 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":415,"threads_started":1}
I20260924 03:21:19.128775 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:19.112405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:41721 (local address 127.10.12.190:43383)
0924 03:21:19.112614 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:19.112621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:19.113652 (+ 1031us) server_negotiation.cc:407] Connection header received
0924 03:21:19.113948 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:19.113955 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:19.114041 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:19.114135 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:19.115499 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.116508 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.118093 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.118578 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.118719 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:19.121258 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:19.121265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:19.121297 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:19.121343 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:19.124947 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:19.125849 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:19.125856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:19.125860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:19.126047 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:19.126642 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:19.126646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:19.126650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:19.127047 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:19.127220 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:19.128127 (+ 907us) server_negotiation.cc:299] Negotiation successful
0924 03:21:19.128436 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 03:21:19.128950 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 03:21:19 dist-test-slave-00qs krb5kdc[11371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220079, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 03:21:19 dist-test-slave-00qs krb5kdc[11371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220079, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:19.181955 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:19.164125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52762 (local address 127.10.12.190:43383)
0924 03:21:19.164380 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:19.164388 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:19.164577 (+ 189us) server_negotiation.cc:407] Connection header received
0924 03:21:19.164924 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:19.164929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:19.165016 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:19.165159 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:19.166938 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.167837 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.169389 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.169944 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.170063 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:19.174463 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:19.174471 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:19.174477 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:19.174526 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:19.178133 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:19.179123 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:19.179128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:19.179130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:19.179312 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:19.179882 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:19.179892 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:19.179895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:19.180465 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:19.180768 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:19.181325 (+ 557us) server_negotiation.cc:299] Negotiation successful
0924 03:21:19.181539 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 03:21:19.186902 11415 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:21:19.194170 11415 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:21:19.218986 11915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:19.209338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39752 (local address 127.10.12.130:36641)
0924 03:21:19.209904 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:19.209915 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:19.210628 (+ 713us) server_negotiation.cc:407] Connection header received
0924 03:21:19.211022 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:19.211031 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:19.211321 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:19.211514 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:19.212342 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.213814 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.216642 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.217799 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.217933 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:19.218169 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:19.218479 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 03:21:19.218674 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":255,"threads_started":1}
I20260924 03:21:19.219038 11914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:19.208865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:36641 (local address 127.0.0.1:39752)
0924 03:21:19.210466 (+ 1601us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:19.210517 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:19.210798 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:19.211657 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:19.211665 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:19.211694 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:19.212116 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:19.212129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.213980 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.213987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:19.216441 (+ 2454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:19.216456 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.218028 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.218035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:19.218241 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:19.218291 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 03:21:19.218411 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 03:21:19.218630 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1345,"mutex_wait_us":1,"thread_start_us":1187,"threads_started":1}
I20260924 03:21:19.221101 11916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:19.209287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:44489 (local address 127.0.0.1:55886)
0924 03:21:19.211383 (+ 2096us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:19.211401 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:19.211504 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:19.213388 (+ 1884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:19.213393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:19.213424 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:19.213812 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:19.213824 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.215717 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.215723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:19.217968 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:19.217984 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.220227 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.220234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:19.220408 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:19.220442 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 03:21:19.220527 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 03:21:19.220684 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1933,"thread_start_us":1782,"threads_started":1}
I20260924 03:21:19.222354 11917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:19.209571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55886 (local address 127.10.12.129:44489)
0924 03:21:19.212735 (+ 3164us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:19.212744 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:19.212774 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:21:19.212867 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:19.212875 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:19.213149 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:19.213367 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:19.214033 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.215562 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.218167 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.219975 (+ 1808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.220132 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:19.221803 (+ 1671us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:19.221936 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 03:21:19.222132 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3008,"thread_start_us":2875,"threads_started":1}
I20260924 03:21:19.223519 11692 tablet_service.cc:1467] Processing CreateTablet for tablet da9c2b0e6bf14564a9b9f8a6838c94a1 (DEFAULT_TABLE table=test-table [id=e4ee5c5b1ce947d88aeff56df9765ffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:19.224303 11918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:19.211529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:40969 (local address 127.0.0.1:55572)
0924 03:21:19.215321 (+ 3792us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:19.215342 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:19.215484 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:19.216489 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:19.216494 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:19.216521 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:19.216962 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:19.216975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.219038 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.219052 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:19.221514 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:19.221530 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.223460 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.223467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:19.223647 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:19.223682 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 03:21:19.223788 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 03:21:19.223935 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3653,"thread_start_us":3505,"threads_started":1}
I20260924 03:21:19.224406 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:19.212298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55572 (local address 127.10.12.131:40969)
0924 03:21:19.214880 (+ 2582us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:19.214888 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:19.215461 (+ 573us) server_negotiation.cc:407] Connection header received
0924 03:21:19.215627 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:19.215636 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:19.215929 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:19.216164 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:19.217205 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.218856 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.221721 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.223132 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.223278 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:19.223508 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:19.223857 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 03:21:19.224061 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2399,"thread_start_us":1362,"threads_started":1}
I20260924 03:21:19.225234 11692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9c2b0e6bf14564a9b9f8a6838c94a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:19.226063 11553 tablet_service.cc:1467] Processing CreateTablet for tablet da9c2b0e6bf14564a9b9f8a6838c94a1 (DEFAULT_TABLE table=test-table [id=e4ee5c5b1ce947d88aeff56df9765ffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:19.227347 11553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9c2b0e6bf14564a9b9f8a6838c94a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:19.228057 11831 tablet_service.cc:1467] Processing CreateTablet for tablet da9c2b0e6bf14564a9b9f8a6838c94a1 (DEFAULT_TABLE table=test-table [id=e4ee5c5b1ce947d88aeff56df9765ffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:19.229549 11831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9c2b0e6bf14564a9b9f8a6838c94a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:19.239377 11920 tablet_bootstrap.cc:492] T da9c2b0e6bf14564a9b9f8a6838c94a1 P 1c5ef5580b16496bad3459d19814ccd7: Bootstrap starting.
I20260924 03:21:19.243314 11920 tablet_bootstrap.cc:654] T da9c2b0e6bf14564a9b9f8a6838c94a1 P 1c5ef5580b16496bad3459d19814ccd7: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:19.244544 11920 log.cc:824] T da9c2b0e6bf14564a9b9f8a6838c94a1 P 1c5ef5580b16496bad3459d19814ccd7: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:19.247586 11920 tablet_bootstrap.cc:492] T da9c2b0e6bf14564a9b9f8a6838c94a1 P 1c5ef5580b16496bad3459d19814ccd7: No bootstrap required, opened a new log
I20260924 03:21:19.247866 11920 ts_tablet_manager.cc:1397] T da9c2b0e6bf14564a9b9f8a6838c94a1 P 1c5ef5580b16496bad3459d19814ccd7: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 03:21:19.248202 11921 tablet_bootstrap.cc:492] T da9c2b0e6bf14564a9b9f8a6838c94a1 P fcd549f721904765b980cba6e5fd4046: Bootstrap starting.
I20260924 03:21:19.252894 11921 tablet_bootstrap.cc:654] T da9c2b0e6bf14564a9b9f8a6838c94a1 P fcd549f721904765b980cba6e5fd4046: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:19.253294 11923 tablet_bootstrap.cc:492] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b: Bootstrap starting.
I20260924 03:21:19.254355 11921 log.cc:824] T da9c2b0e6bf14564a9b9f8a6838c94a1 P fcd549f721904765b980cba6e5fd4046: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:19.256906 11920 raft_consensus.cc:348] T da9c2b0e6bf14564a9b9f8a6838c94a1 P 1c5ef5580b16496bad3459d19814ccd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:19.257323 11920 raft_consensus.cc:374] T da9c2b0e6bf14564a9b9f8a6838c94a1 P 1c5ef5580b16496bad3459d19814ccd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:19.257421 11920 raft_consensus.cc:729] T da9c2b0e6bf14564a9b9f8a6838c94a1 P 1c5ef5580b16496bad3459d19814ccd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c5ef5580b16496bad3459d19814ccd7, State: Initialized, Role: FOLLOWER
I20260924 03:21:19.258136 11923 tablet_bootstrap.cc:654] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:19.258100 11920 consensus_queue.cc:260] T da9c2b0e6bf14564a9b9f8a6838c94a1 P 1c5ef5580b16496bad3459d19814ccd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:19.259312 11921 tablet_bootstrap.cc:492] T da9c2b0e6bf14564a9b9f8a6838c94a1 P fcd549f721904765b980cba6e5fd4046: No bootstrap required, opened a new log
I20260924 03:21:19.259657 11921 ts_tablet_manager.cc:1397] T da9c2b0e6bf14564a9b9f8a6838c94a1 P fcd549f721904765b980cba6e5fd4046: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260924 03:21:19.260110 11920 ts_tablet_manager.cc:1428] T da9c2b0e6bf14564a9b9f8a6838c94a1 P 1c5ef5580b16496bad3459d19814ccd7: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260924 03:21:19.261580 11923 log.cc:824] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:19.265555 11923 tablet_bootstrap.cc:492] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b: No bootstrap required, opened a new log
I20260924 03:21:19.266062 11923 ts_tablet_manager.cc:1397] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260924 03:21:19.270500 11921 raft_consensus.cc:348] T da9c2b0e6bf14564a9b9f8a6838c94a1 P fcd549f721904765b980cba6e5fd4046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:19.271167 11921 raft_consensus.cc:374] T da9c2b0e6bf14564a9b9f8a6838c94a1 P fcd549f721904765b980cba6e5fd4046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:19.271276 11921 raft_consensus.cc:729] T da9c2b0e6bf14564a9b9f8a6838c94a1 P fcd549f721904765b980cba6e5fd4046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd549f721904765b980cba6e5fd4046, State: Initialized, Role: FOLLOWER
I20260924 03:21:19.272075 11921 consensus_queue.cc:260] T da9c2b0e6bf14564a9b9f8a6838c94a1 P fcd549f721904765b980cba6e5fd4046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:19.274081 11762 heartbeater.cc:503] Master 127.10.12.190:43383 was elected leader, sending a full tablet report...
I20260924 03:21:19.274780 11921 ts_tablet_manager.cc:1428] T da9c2b0e6bf14564a9b9f8a6838c94a1 P fcd549f721904765b980cba6e5fd4046: Time spent starting tablet: real 0.015s user 0.007s sys 0.010s
I20260924 03:21:19.277704 11923 raft_consensus.cc:348] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:19.278131 11923 raft_consensus.cc:374] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:19.278226 11923 raft_consensus.cc:729] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6759c961f724ee1ab5dd4bc22b3411b, State: Initialized, Role: FOLLOWER
I20260924 03:21:19.278904 11923 consensus_queue.cc:260] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:19.280120 11901 heartbeater.cc:503] Master 127.10.12.190:43383 was elected leader, sending a full tablet report...
I20260924 03:21:19.280768 11923 ts_tablet_manager.cc:1428] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260924 03:21:19.294525 11624 tablet.cc:2369] T da9c2b0e6bf14564a9b9f8a6838c94a1 P 1c5ef5580b16496bad3459d19814ccd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:21:19.344772 11763 tablet.cc:2369] T da9c2b0e6bf14564a9b9f8a6838c94a1 P fcd549f721904765b980cba6e5fd4046: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:21:19.372272 11902 tablet.cc:2369] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:21:19.418797 11928 raft_consensus.cc:482] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:21:19.419418 11928 raft_consensus.cc:504] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:19.421315 11928 leader_election.cc:288] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcd549f721904765b980cba6e5fd4046 (127.10.12.130:36641), 1c5ef5580b16496bad3459d19814ccd7 (127.10.12.129:44489)
I20260924 03:21:19.430557 11917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:19.421922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:49491 (local address 127.10.12.129:44489)
0924 03:21:19.422127 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:19.422133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:19.423412 (+ 1279us) server_negotiation.cc:407] Connection header received
0924 03:21:19.423907 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:19.423915 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:19.424051 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:19.424143 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:19.424730 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.425787 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.427936 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.428845 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.429054 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:19.429346 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:19.430131 (+ 785us) server_negotiation.cc:299] Negotiation successful
0924 03:21:19.430290 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 03:21:19.430828 11929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:19.422383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:44489 (local address 127.10.12.131:49491)
0924 03:21:19.423205 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:19.423220 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:19.423551 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:19.424185 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:19.424191 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:19.424224 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:19.424553 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:19.424564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.425960 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.425966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:19.427768 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:19.427780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.429285 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.429292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:19.429639 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:19.429863 (+ 224us) client_negotiation.cc:769] Sending connection context
0924 03:21:19.430047 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 03:21:19.430376 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":88,"thread_start_us":521,"threads_started":1}
I20260924 03:21:19.431205 11790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:19.422148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:36641 (local address 127.10.12.131:57865)
0924 03:21:19.422463 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:19.422485 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:19.422657 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:19.423192 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:19.423197 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:19.423225 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:19.423458 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:19.423471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.425091 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.425098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:19.427263 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:19.427282 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.429045 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.429051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:19.429681 (+ 630us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:19.429942 (+ 261us) client_negotiation.cc:769] Sending connection context
0924 03:21:19.430081 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 03:21:19.430290 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":214}
I20260924 03:21:19.431504 11915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:19.421837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:57865 (local address 127.10.12.130:36641)
0924 03:21:19.422082 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:19.422089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:19.422784 (+ 695us) server_negotiation.cc:407] Connection header received
0924 03:21:19.422870 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:19.422877 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:19.422952 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:19.423026 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:19.423739 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.424912 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.427480 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.428811 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.428989 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:19.429068 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:19.430494 (+ 1426us) server_negotiation.cc:299] Negotiation successful
0924 03:21:19.431054 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 03:21:19.432097 11573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9c2b0e6bf14564a9b9f8a6838c94a1" candidate_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c5ef5580b16496bad3459d19814ccd7" is_pre_election: true
I20260924 03:21:19.432391 11717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9c2b0e6bf14564a9b9f8a6838c94a1" candidate_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcd549f721904765b980cba6e5fd4046" is_pre_election: true
I20260924 03:21:19.432693 11573 raft_consensus.cc:2454] T da9c2b0e6bf14564a9b9f8a6838c94a1 P 1c5ef5580b16496bad3459d19814ccd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6759c961f724ee1ab5dd4bc22b3411b in term 0.
I20260924 03:21:19.432888 11717 raft_consensus.cc:2454] T da9c2b0e6bf14564a9b9f8a6838c94a1 P fcd549f721904765b980cba6e5fd4046 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6759c961f724ee1ab5dd4bc22b3411b in term 0.
I20260924 03:21:19.433504 11782 leader_election.cc:302] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c5ef5580b16496bad3459d19814ccd7, e6759c961f724ee1ab5dd4bc22b3411b; no voters:
I20260924 03:21:19.433931 11928 raft_consensus.cc:2781] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:21:19.434077 11928 raft_consensus.cc:482] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:21:19.434176 11928 raft_consensus.cc:3037] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:19.436703 11928 raft_consensus.cc:504] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:19.437351 11928 leader_election.cc:288] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b [CANDIDATE]: Term 1 election: Requested vote from peers fcd549f721904765b980cba6e5fd4046 (127.10.12.130:36641), 1c5ef5580b16496bad3459d19814ccd7 (127.10.12.129:44489)
I20260924 03:21:19.437961 11717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9c2b0e6bf14564a9b9f8a6838c94a1" candidate_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcd549f721904765b980cba6e5fd4046"
I20260924 03:21:19.437968 11573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9c2b0e6bf14564a9b9f8a6838c94a1" candidate_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c5ef5580b16496bad3459d19814ccd7"
I20260924 03:21:19.438158 11717 raft_consensus.cc:3037] T da9c2b0e6bf14564a9b9f8a6838c94a1 P fcd549f721904765b980cba6e5fd4046 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:19.438169 11573 raft_consensus.cc:3037] T da9c2b0e6bf14564a9b9f8a6838c94a1 P 1c5ef5580b16496bad3459d19814ccd7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:19.440579 11717 raft_consensus.cc:2454] T da9c2b0e6bf14564a9b9f8a6838c94a1 P fcd549f721904765b980cba6e5fd4046 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6759c961f724ee1ab5dd4bc22b3411b in term 1.
I20260924 03:21:19.440580 11573 raft_consensus.cc:2454] T da9c2b0e6bf14564a9b9f8a6838c94a1 P 1c5ef5580b16496bad3459d19814ccd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6759c961f724ee1ab5dd4bc22b3411b in term 1.
I20260924 03:21:19.441082 11785 leader_election.cc:302] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6759c961f724ee1ab5dd4bc22b3411b, fcd549f721904765b980cba6e5fd4046; no voters:
I20260924 03:21:19.441430 11928 raft_consensus.cc:2781] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:19.441892 11928 raft_consensus.cc:686] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b [term 1 LEADER]: Becoming Leader. State: Replica: e6759c961f724ee1ab5dd4bc22b3411b, State: Running, Role: LEADER
I20260924 03:21:19.442418 11928 consensus_queue.cc:237] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:19.447736 11415 catalog_manager.cc:5495] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b reported cstate change: term changed from 0 to 1, leader changed from <none> to e6759c961f724ee1ab5dd4bc22b3411b (127.10.12.131). New cstate: current_term: 1 leader_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } health_report { overall_health: HEALTHY } } }
I20260924 03:21:19.474560 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:19.468736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.10.12.131:40969)
0924 03:21:19.468995 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:19.469001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:19.469099 (+ 98us) server_negotiation.cc:407] Connection header received
0924 03:21:19.469485 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:19.469489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:19.469571 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:19.469693 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:19.470073 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.470815 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.472247 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.472657 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.472769 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:19.473322 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:19.473915 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:19.474045 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:19.474202 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 03:21:19.474292 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 03:21:19.483332 11573 raft_consensus.cc:1261] T da9c2b0e6bf14564a9b9f8a6838c94a1 P 1c5ef5580b16496bad3459d19814ccd7 [term 1 FOLLOWER]: Refusing update from remote peer e6759c961f724ee1ab5dd4bc22b3411b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:21:19.483332 11717 raft_consensus.cc:1261] T da9c2b0e6bf14564a9b9f8a6838c94a1 P fcd549f721904765b980cba6e5fd4046 [term 1 FOLLOWER]: Refusing update from remote peer e6759c961f724ee1ab5dd4bc22b3411b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:21:19.484354 11928 consensus_queue.cc:1035] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:21:19.484684 11930 consensus_queue.cc:1035] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:21:19.519767 11416 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:21:19.520188 11416 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:21:19.524737 11416 catalog_manager.cc:5782] T 00000000000000000000000000000000 P add047dafdab4e0fbbedef99257fca6b: Sending DeleteTablet for 3 replicas of tablet da9c2b0e6bf14564a9b9f8a6838c94a1
I20260924 03:21:19.526264 11553 tablet_service.cc:1514] Processing DeleteTablet for tablet da9c2b0e6bf14564a9b9f8a6838c94a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:21:19 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:55886
I20260924 03:21:19.526268 11692 tablet_service.cc:1514] Processing DeleteTablet for tablet da9c2b0e6bf14564a9b9f8a6838c94a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:21:19 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:39752
I20260924 03:21:19.526469 11831 tablet_service.cc:1514] Processing DeleteTablet for tablet da9c2b0e6bf14564a9b9f8a6838c94a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:21:19 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:55572
I20260924 03:21:19.527858 11415 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:21:19.528744 11415 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:21:19.530695 11944 tablet_replica.cc:330] T da9c2b0e6bf14564a9b9f8a6838c94a1 P 1c5ef5580b16496bad3459d19814ccd7: stopping tablet replica
I20260924 03:21:19.531196 11944 raft_consensus.cc:2229] T da9c2b0e6bf14564a9b9f8a6838c94a1 P 1c5ef5580b16496bad3459d19814ccd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:21:19.531764 11944 raft_consensus.cc:2258] T da9c2b0e6bf14564a9b9f8a6838c94a1 P 1c5ef5580b16496bad3459d19814ccd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:21:19.533504 11943 tablet_replica.cc:330] T da9c2b0e6bf14564a9b9f8a6838c94a1 P fcd549f721904765b980cba6e5fd4046: stopping tablet replica
I20260924 03:21:19.533565 11945 tablet_replica.cc:330] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b: stopping tablet replica
I20260924 03:21:19.534426 11943 raft_consensus.cc:2229] T da9c2b0e6bf14564a9b9f8a6838c94a1 P fcd549f721904765b980cba6e5fd4046 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:21:19.534437 11945 raft_consensus.cc:2229] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:21:19.535223 11943 raft_consensus.cc:2258] T da9c2b0e6bf14564a9b9f8a6838c94a1 P fcd549f721904765b980cba6e5fd4046 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:21:19.535604 11945 raft_consensus.cc:2258] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:21:19.537598 11944 ts_tablet_manager.cc:1905] T da9c2b0e6bf14564a9b9f8a6838c94a1 P 1c5ef5580b16496bad3459d19814ccd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:21:19.537865 11945 ts_tablet_manager.cc:1905] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:21:19.540584 11692 tablet_service.cc:1467] Processing CreateTablet for tablet 72dbc4568e98414b82bc62957a1b00d8 (DEFAULT_TABLE table=test-table [id=a21b79e8b85345479c23fee3a9492a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:19.540752 11553 tablet_service.cc:1467] Processing CreateTablet for tablet 72dbc4568e98414b82bc62957a1b00d8 (DEFAULT_TABLE table=test-table [id=a21b79e8b85345479c23fee3a9492a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:19.541076 11692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72dbc4568e98414b82bc62957a1b00d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:19.541237 11553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72dbc4568e98414b82bc62957a1b00d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:19.542208 11831 tablet_service.cc:1467] Processing CreateTablet for tablet 72dbc4568e98414b82bc62957a1b00d8 (DEFAULT_TABLE table=test-table [id=a21b79e8b85345479c23fee3a9492a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:19.542662 11831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72dbc4568e98414b82bc62957a1b00d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:19.545399 11943 ts_tablet_manager.cc:1905] T da9c2b0e6bf14564a9b9f8a6838c94a1 P fcd549f721904765b980cba6e5fd4046: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:21:19.547746 11920 tablet_bootstrap.cc:492] T 72dbc4568e98414b82bc62957a1b00d8 P 1c5ef5580b16496bad3459d19814ccd7: Bootstrap starting.
I20260924 03:21:19.548218 11921 tablet_bootstrap.cc:492] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046: Bootstrap starting.
I20260924 03:21:19.549952 11920 tablet_bootstrap.cc:654] T 72dbc4568e98414b82bc62957a1b00d8 P 1c5ef5580b16496bad3459d19814ccd7: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:19.550096 11921 tablet_bootstrap.cc:654] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:19.553292 11944 ts_tablet_manager.cc:1918] T da9c2b0e6bf14564a9b9f8a6838c94a1 P 1c5ef5580b16496bad3459d19814ccd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:21:19.553494 11944 log.cc:1196] T da9c2b0e6bf14564a9b9f8a6838c94a1 P 1c5ef5580b16496bad3459d19814ccd7: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-0/wal/wals/da9c2b0e6bf14564a9b9f8a6838c94a1
I20260924 03:21:19.554237 11923 tablet_bootstrap.cc:492] T 72dbc4568e98414b82bc62957a1b00d8 P e6759c961f724ee1ab5dd4bc22b3411b: Bootstrap starting.
I20260924 03:21:19.556229 11923 tablet_bootstrap.cc:654] T 72dbc4568e98414b82bc62957a1b00d8 P e6759c961f724ee1ab5dd4bc22b3411b: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:19.563740 11921 tablet_bootstrap.cc:492] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046: No bootstrap required, opened a new log
I20260924 03:21:19.563781 11920 tablet_bootstrap.cc:492] T 72dbc4568e98414b82bc62957a1b00d8 P 1c5ef5580b16496bad3459d19814ccd7: No bootstrap required, opened a new log
I20260924 03:21:19.563979 11920 ts_tablet_manager.cc:1397] T 72dbc4568e98414b82bc62957a1b00d8 P 1c5ef5580b16496bad3459d19814ccd7: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.004s
I20260924 03:21:19.564138 11921 ts_tablet_manager.cc:1397] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.008s
I20260924 03:21:19.564996 11920 raft_consensus.cc:348] T 72dbc4568e98414b82bc62957a1b00d8 P 1c5ef5580b16496bad3459d19814ccd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:19.565256 11920 raft_consensus.cc:374] T 72dbc4568e98414b82bc62957a1b00d8 P 1c5ef5580b16496bad3459d19814ccd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:19.565299 11943 ts_tablet_manager.cc:1918] T da9c2b0e6bf14564a9b9f8a6838c94a1 P fcd549f721904765b980cba6e5fd4046: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:21:19.565351 11920 raft_consensus.cc:729] T 72dbc4568e98414b82bc62957a1b00d8 P 1c5ef5580b16496bad3459d19814ccd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c5ef5580b16496bad3459d19814ccd7, State: Initialized, Role: FOLLOWER
I20260924 03:21:19.565104 11921 raft_consensus.cc:348] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:19.565624 11921 raft_consensus.cc:374] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:19.565498 11943 log.cc:1196] T da9c2b0e6bf14564a9b9f8a6838c94a1 P fcd549f721904765b980cba6e5fd4046: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/da9c2b0e6bf14564a9b9f8a6838c94a1
I20260924 03:21:19.565755 11921 raft_consensus.cc:729] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd549f721904765b980cba6e5fd4046, State: Initialized, Role: FOLLOWER
I20260924 03:21:19.565758 11920 consensus_queue.cc:260] T 72dbc4568e98414b82bc62957a1b00d8 P 1c5ef5580b16496bad3459d19814ccd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:19.566370 11943 ts_tablet_manager.cc:1939] T da9c2b0e6bf14564a9b9f8a6838c94a1 P fcd549f721904765b980cba6e5fd4046: Deleting consensus metadata
I20260924 03:21:19.566354 11920 ts_tablet_manager.cc:1428] T 72dbc4568e98414b82bc62957a1b00d8 P 1c5ef5580b16496bad3459d19814ccd7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:21:19.566226 11921 consensus_queue.cc:260] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:19.567013 11921 ts_tablet_manager.cc:1428] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260924 03:21:19.568029 11944 ts_tablet_manager.cc:1939] T da9c2b0e6bf14564a9b9f8a6838c94a1 P 1c5ef5580b16496bad3459d19814ccd7: Deleting consensus metadata
I20260924 03:21:19.570278 11945 ts_tablet_manager.cc:1918] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:21:19.570614 11945 log.cc:1196] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/da9c2b0e6bf14564a9b9f8a6838c94a1
I20260924 03:21:19.571372 11945 ts_tablet_manager.cc:1939] T da9c2b0e6bf14564a9b9f8a6838c94a1 P e6759c961f724ee1ab5dd4bc22b3411b: Deleting consensus metadata
I20260924 03:21:19.571841 11400 catalog_manager.cc:4841] TS 1c5ef5580b16496bad3459d19814ccd7 (127.10.12.129:44489): tablet da9c2b0e6bf14564a9b9f8a6838c94a1 (table test-table [id=e4ee5c5b1ce947d88aeff56df9765ffe]) successfully deleted
I20260924 03:21:19.572955 11923 tablet_bootstrap.cc:492] T 72dbc4568e98414b82bc62957a1b00d8 P e6759c961f724ee1ab5dd4bc22b3411b: No bootstrap required, opened a new log
I20260924 03:21:19.573141 11923 ts_tablet_manager.cc:1397] T 72dbc4568e98414b82bc62957a1b00d8 P e6759c961f724ee1ab5dd4bc22b3411b: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.000s
I20260924 03:21:19.573969 11400 catalog_manager.cc:4841] TS e6759c961f724ee1ab5dd4bc22b3411b (127.10.12.131:40969): tablet da9c2b0e6bf14564a9b9f8a6838c94a1 (table test-table [id=e4ee5c5b1ce947d88aeff56df9765ffe]) successfully deleted
I20260924 03:21:19.574009 11923 raft_consensus.cc:348] T 72dbc4568e98414b82bc62957a1b00d8 P e6759c961f724ee1ab5dd4bc22b3411b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:19.574254 11923 raft_consensus.cc:374] T 72dbc4568e98414b82bc62957a1b00d8 P e6759c961f724ee1ab5dd4bc22b3411b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:19.574355 11923 raft_consensus.cc:729] T 72dbc4568e98414b82bc62957a1b00d8 P e6759c961f724ee1ab5dd4bc22b3411b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6759c961f724ee1ab5dd4bc22b3411b, State: Initialized, Role: FOLLOWER
I20260924 03:21:19.574656 11923 consensus_queue.cc:260] T 72dbc4568e98414b82bc62957a1b00d8 P e6759c961f724ee1ab5dd4bc22b3411b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:19.576076 11923 ts_tablet_manager.cc:1428] T 72dbc4568e98414b82bc62957a1b00d8 P e6759c961f724ee1ab5dd4bc22b3411b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:21:19.576983 11403 catalog_manager.cc:4841] TS fcd549f721904765b980cba6e5fd4046 (127.10.12.130:36641): tablet da9c2b0e6bf14564a9b9f8a6838c94a1 (table test-table [id=e4ee5c5b1ce947d88aeff56df9765ffe]) successfully deleted
I20260924 03:21:19.644059 11927 raft_consensus.cc:482] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:21:19.644366 11927 raft_consensus.cc:504] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:19.646579 11927 leader_election.cc:288] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c5ef5580b16496bad3459d19814ccd7 (127.10.12.129:44489), e6759c961f724ee1ab5dd4bc22b3411b (127.10.12.131:40969)
I20260924 03:21:19.655789 11917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:19.647538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:58525 (local address 127.10.12.129:44489)
0924 03:21:19.647911 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:19.647938 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:19.648955 (+ 1017us) server_negotiation.cc:407] Connection header received
0924 03:21:19.649210 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:19.649217 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:19.649310 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:19.649381 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:19.650353 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.651346 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.653710 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.654586 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.654720 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:19.654782 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:19.655379 (+ 597us) server_negotiation.cc:299] Negotiation successful
0924 03:21:19.655523 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260924 03:21:19.655891 11949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:19.648261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:44489 (local address 127.10.12.130:58525)
0924 03:21:19.648848 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:19.648867 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:19.649008 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:19.649620 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:19.649624 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:19.649654 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:19.650163 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:19.650176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.651519 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.651525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:19.653481 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:19.653499 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.654971 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.654976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:19.655199 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:19.655223 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 03:21:19.655318 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 03:21:19.655476 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":45,"thread_start_us":292,"threads_started":1}
I20260924 03:21:19.657111 11573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72dbc4568e98414b82bc62957a1b00d8" candidate_uuid: "fcd549f721904765b980cba6e5fd4046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c5ef5580b16496bad3459d19814ccd7" is_pre_election: true
I20260924 03:21:19.657406 11573 raft_consensus.cc:2454] T 72dbc4568e98414b82bc62957a1b00d8 P 1c5ef5580b16496bad3459d19814ccd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcd549f721904765b980cba6e5fd4046 in term 0.
I20260924 03:21:19.657933 11643 leader_election.cc:302] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c5ef5580b16496bad3459d19814ccd7, fcd549f721904765b980cba6e5fd4046; no voters:
I20260924 03:21:19.658502 11927 raft_consensus.cc:2781] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:21:19.658646 11927 raft_consensus.cc:482] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:21:19.658756 11927 raft_consensus.cc:3037] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:19.659296 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:19.649214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:38019 (local address 127.10.12.131:40969)
0924 03:21:19.649402 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:19.649408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:19.650370 (+ 962us) server_negotiation.cc:407] Connection header received
0924 03:21:19.650600 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:19.650605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:19.650698 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:19.650826 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:19.651429 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.652577 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.655665 (+ 3088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.657647 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.657817 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:19.657901 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:19.658771 (+ 870us) server_negotiation.cc:299] Negotiation successful
0924 03:21:19.658941 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 03:21:19.660348 11950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:19.649015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:40969 (local address 127.10.12.130:38019)
0924 03:21:19.650244 (+ 1229us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:19.650265 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:19.650458 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:19.650846 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:19.650852 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:19.650879 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:19.651254 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:19.651267 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.652742 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.652766 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:19.655296 (+ 2530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:19.655333 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.658166 (+ 2833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.658173 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:19.658521 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:19.658562 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 03:21:19.659577 (+ 1015us) client_negotiation.cc:240] Negotiation successful
0924 03:21:19.659861 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1083,"mutex_wait_us":1,"thread_start_us":844,"threads_started":1}
I20260924 03:21:19.661387 11851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72dbc4568e98414b82bc62957a1b00d8" candidate_uuid: "fcd549f721904765b980cba6e5fd4046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" is_pre_election: true
I20260924 03:21:19.661871 11851 raft_consensus.cc:2454] T 72dbc4568e98414b82bc62957a1b00d8 P e6759c961f724ee1ab5dd4bc22b3411b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcd549f721904765b980cba6e5fd4046 in term 0.
I20260924 03:21:19.661828 11927 raft_consensus.cc:504] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:19.662580 11927 leader_election.cc:288] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046 [CANDIDATE]: Term 1 election: Requested vote from peers 1c5ef5580b16496bad3459d19814ccd7 (127.10.12.129:44489), e6759c961f724ee1ab5dd4bc22b3411b (127.10.12.131:40969)
I20260924 03:21:19.663553 11573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72dbc4568e98414b82bc62957a1b00d8" candidate_uuid: "fcd549f721904765b980cba6e5fd4046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c5ef5580b16496bad3459d19814ccd7"
I20260924 03:21:19.663609 11851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72dbc4568e98414b82bc62957a1b00d8" candidate_uuid: "fcd549f721904765b980cba6e5fd4046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6759c961f724ee1ab5dd4bc22b3411b"
I20260924 03:21:19.663865 11851 raft_consensus.cc:3037] T 72dbc4568e98414b82bc62957a1b00d8 P e6759c961f724ee1ab5dd4bc22b3411b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:19.663785 11573 raft_consensus.cc:3037] T 72dbc4568e98414b82bc62957a1b00d8 P 1c5ef5580b16496bad3459d19814ccd7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:19.666225 11573 raft_consensus.cc:2454] T 72dbc4568e98414b82bc62957a1b00d8 P 1c5ef5580b16496bad3459d19814ccd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcd549f721904765b980cba6e5fd4046 in term 1.
I20260924 03:21:19.666777 11643 leader_election.cc:302] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c5ef5580b16496bad3459d19814ccd7, fcd549f721904765b980cba6e5fd4046; no voters:
I20260924 03:21:19.667075 11927 raft_consensus.cc:2781] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:19.667245 11851 raft_consensus.cc:2454] T 72dbc4568e98414b82bc62957a1b00d8 P e6759c961f724ee1ab5dd4bc22b3411b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcd549f721904765b980cba6e5fd4046 in term 1.
I20260924 03:21:19.667783 11927 raft_consensus.cc:686] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046 [term 1 LEADER]: Becoming Leader. State: Replica: fcd549f721904765b980cba6e5fd4046, State: Running, Role: LEADER
I20260924 03:21:19.668301 11927 consensus_queue.cc:237] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:19.672627 11416 catalog_manager.cc:5495] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046 reported cstate change: term changed from 0 to 1, leader changed from <none> to fcd549f721904765b980cba6e5fd4046 (127.10.12.130). New cstate: current_term: 1 leader_uuid: "fcd549f721904765b980cba6e5fd4046" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } health_report { overall_health: UNKNOWN } } }
Sep 24 03:21:19 dist-test-slave-00qs krb5kdc[11371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220077, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:19.759011 11958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:19.742678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:43383 (local address 127.0.0.1:48786)
0924 03:21:19.743206 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:19.743234 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:19.743407 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:19.743825 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:19.743829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:19.744542 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:19.744739 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:19.744749 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.746138 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.746144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:19.747626 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:19.747647 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.748468 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.748475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:19.748652 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:19.749361 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:19.749380 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:19.752164 (+ 2784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:19.756125 (+ 3961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:19.756145 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:19.756155 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:19.756739 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:19.757218 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:19.757221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:19.757223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:19.757384 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:19.757938 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:19.757945 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:19.758112 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 03:21:19.758258 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 03:21:19.758568 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":262,"threads_started":1}
I20260924 03:21:19.759111 11959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:19.742739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48786 (local address 127.10.12.190:43383)
0924 03:21:19.743561 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:19.743565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:19.743578 (+ 13us) server_negotiation.cc:407] Connection header received
0924 03:21:19.743617 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:19.743620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:19.743672 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:19.743762 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:19.744897 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.745791 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.747788 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:19.748290 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:19.748416 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:19.752403 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:19.752409 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:19.752412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:19.752448 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:19.755791 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:19.756948 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:19.756954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:19.756957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:19.757104 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:19.757502 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:19.757507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:19.757510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:19.757800 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:19.757957 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:19.758386 (+ 429us) server_negotiation.cc:299] Negotiation successful
0924 03:21:19.758752 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":707,"thread_start_us":573,"threads_started":1}
I20260924 03:21:19.765465 11416 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:48786:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 03:21:19.766561 11416 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:21:19.774991 11553 tablet_service.cc:1467] Processing CreateTablet for tablet cbdf3df0e269498c8bb57bb9badf204f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ed6d008339244f29a3ec2f0bec0785a1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:21:19.775087 11831 tablet_service.cc:1467] Processing CreateTablet for tablet cbdf3df0e269498c8bb57bb9badf204f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ed6d008339244f29a3ec2f0bec0785a1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:21:19.775606 11831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbdf3df0e269498c8bb57bb9badf204f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:19.775607 11553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbdf3df0e269498c8bb57bb9badf204f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:19.776928 11692 tablet_service.cc:1467] Processing CreateTablet for tablet cbdf3df0e269498c8bb57bb9badf204f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ed6d008339244f29a3ec2f0bec0785a1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:21:19.777474 11692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbdf3df0e269498c8bb57bb9badf204f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:19.782974 11920 tablet_bootstrap.cc:492] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7: Bootstrap starting.
I20260924 03:21:19.783433 11923 tablet_bootstrap.cc:492] T cbdf3df0e269498c8bb57bb9badf204f P e6759c961f724ee1ab5dd4bc22b3411b: Bootstrap starting.
I20260924 03:21:19.784807 11920 tablet_bootstrap.cc:654] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:19.785192 11923 tablet_bootstrap.cc:654] T cbdf3df0e269498c8bb57bb9badf204f P e6759c961f724ee1ab5dd4bc22b3411b: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:19.785760 11921 tablet_bootstrap.cc:492] T cbdf3df0e269498c8bb57bb9badf204f P fcd549f721904765b980cba6e5fd4046: Bootstrap starting.
I20260924 03:21:19.787873 11921 tablet_bootstrap.cc:654] T cbdf3df0e269498c8bb57bb9badf204f P fcd549f721904765b980cba6e5fd4046: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:19.797497 11920 tablet_bootstrap.cc:492] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7: No bootstrap required, opened a new log
I20260924 03:21:19.797704 11920 ts_tablet_manager.cc:1397] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.005s
I20260924 03:21:19.798981 11920 raft_consensus.cc:348] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:19.799269 11920 raft_consensus.cc:374] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:19.799350 11920 raft_consensus.cc:729] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c5ef5580b16496bad3459d19814ccd7, State: Initialized, Role: FOLLOWER
I20260924 03:21:19.799681 11920 consensus_queue.cc:260] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:19.800288 11925 tablet_replica.cc:439] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } } }
I20260924 03:21:19.800642 11925 tablet_replica.cc:442] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:21:19.800557 11921 tablet_bootstrap.cc:492] T cbdf3df0e269498c8bb57bb9badf204f P fcd549f721904765b980cba6e5fd4046: No bootstrap required, opened a new log
I20260924 03:21:19.800921 11921 ts_tablet_manager.cc:1397] T cbdf3df0e269498c8bb57bb9badf204f P fcd549f721904765b980cba6e5fd4046: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.001s
I20260924 03:21:19.802352 11921 raft_consensus.cc:348] T cbdf3df0e269498c8bb57bb9badf204f P fcd549f721904765b980cba6e5fd4046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:19.802556 11923 tablet_bootstrap.cc:492] T cbdf3df0e269498c8bb57bb9badf204f P e6759c961f724ee1ab5dd4bc22b3411b: No bootstrap required, opened a new log
I20260924 03:21:19.802690 11921 raft_consensus.cc:374] T cbdf3df0e269498c8bb57bb9badf204f P fcd549f721904765b980cba6e5fd4046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:19.802738 11923 ts_tablet_manager.cc:1397] T cbdf3df0e269498c8bb57bb9badf204f P e6759c961f724ee1ab5dd4bc22b3411b: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.010s
I20260924 03:21:19.802928 11921 raft_consensus.cc:729] T cbdf3df0e269498c8bb57bb9badf204f P fcd549f721904765b980cba6e5fd4046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd549f721904765b980cba6e5fd4046, State: Initialized, Role: FOLLOWER
I20260924 03:21:19.803282 11921 consensus_queue.cc:260] T cbdf3df0e269498c8bb57bb9badf204f P fcd549f721904765b980cba6e5fd4046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:19.803485 11920 ts_tablet_manager.cc:1428] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260924 03:21:19.803552 11923 raft_consensus.cc:348] T cbdf3df0e269498c8bb57bb9badf204f P e6759c961f724ee1ab5dd4bc22b3411b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:19.803818 11923 raft_consensus.cc:374] T cbdf3df0e269498c8bb57bb9badf204f P e6759c961f724ee1ab5dd4bc22b3411b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:19.803905 11923 raft_consensus.cc:729] T cbdf3df0e269498c8bb57bb9badf204f P e6759c961f724ee1ab5dd4bc22b3411b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6759c961f724ee1ab5dd4bc22b3411b, State: Initialized, Role: FOLLOWER
I20260924 03:21:19.804073 11927 tablet_replica.cc:439] T cbdf3df0e269498c8bb57bb9badf204f P fcd549f721904765b980cba6e5fd4046: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } } }
I20260924 03:21:19.804373 11927 tablet_replica.cc:442] T cbdf3df0e269498c8bb57bb9badf204f P fcd549f721904765b980cba6e5fd4046: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:21:19.804224 11923 consensus_queue.cc:260] T cbdf3df0e269498c8bb57bb9badf204f P e6759c961f724ee1ab5dd4bc22b3411b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:19.805012 11921 ts_tablet_manager.cc:1428] T cbdf3df0e269498c8bb57bb9badf204f P fcd549f721904765b980cba6e5fd4046: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 03:21:19.804900 11938 tablet_replica.cc:439] T cbdf3df0e269498c8bb57bb9badf204f P e6759c961f724ee1ab5dd4bc22b3411b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } } }
I20260924 03:21:19.805279 11938 tablet_replica.cc:442] T cbdf3df0e269498c8bb57bb9badf204f P e6759c961f724ee1ab5dd4bc22b3411b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:21:19.806864 11923 ts_tablet_manager.cc:1428] T cbdf3df0e269498c8bb57bb9badf204f P e6759c961f724ee1ab5dd4bc22b3411b: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260924 03:21:20.011467 11925 raft_consensus.cc:482] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:21:20.011689 11925 raft_consensus.cc:504] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:20.013208 11925 leader_election.cc:288] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcd549f721904765b980cba6e5fd4046 (127.10.12.130:36641), e6759c961f724ee1ab5dd4bc22b3411b (127.10.12.131:40969)
I20260924 03:21:20.022229 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:20.014027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:50009 (local address 127.10.12.130:36641)
0924 03:21:20.014848 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:20.014852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:20.014870 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:21:20.014928 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:20.014932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:20.014991 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:20.015077 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:20.015820 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.016865 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.019609 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.020906 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.021047 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:20.021118 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:20.021613 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 03:21:20.021904 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":624,"threads_started":1}
I20260924 03:21:20.022308 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:20.013836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:36641 (local address 127.10.12.129:50009)
0924 03:21:20.014575 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:20.014594 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:20.014677 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:20.015124 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:20.015128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:20.015167 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:20.015648 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:20.015658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.016988 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.016993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:20.019398 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:20.019415 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.021155 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.021161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:20.021399 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:20.021458 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 03:21:20.021562 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 03:21:20.021753 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":174,"thread_start_us":225,"threads_started":1}
I20260924 03:21:20.022614 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:20.013620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:40969 (local address 127.10.12.129:59081)
0924 03:21:20.014287 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:20.014307 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:20.014522 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:20.015005 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:20.015008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:20.015027 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:20.015435 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:20.015448 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.016899 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.016903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:20.019397 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:20.019415 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.021118 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.021125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:20.021361 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:20.021454 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 03:21:20.021562 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 03:21:20.021857 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":289,"thread_start_us":750,"threads_started":1}
I20260924 03:21:20.022912 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:20.013681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:59081 (local address 127.10.12.131:40969)
0924 03:21:20.013945 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:20.013952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:20.014401 (+ 449us) server_negotiation.cc:407] Connection header received
0924 03:21:20.014697 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:20.014703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:20.014779 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:20.014842 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:20.015632 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.016734 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.019604 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.020857 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.021042 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:20.021118 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:20.021613 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 03:21:20.022673 (+ 1060us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 03:21:20.023087 11717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf3df0e269498c8bb57bb9badf204f" candidate_uuid: "1c5ef5580b16496bad3459d19814ccd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcd549f721904765b980cba6e5fd4046" is_pre_election: true
I20260924 03:21:20.023355 11717 raft_consensus.cc:2454] T cbdf3df0e269498c8bb57bb9badf204f P fcd549f721904765b980cba6e5fd4046 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c5ef5580b16496bad3459d19814ccd7 in term 0.
I20260924 03:21:20.023466 11851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf3df0e269498c8bb57bb9badf204f" candidate_uuid: "1c5ef5580b16496bad3459d19814ccd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" is_pre_election: true
I20260924 03:21:20.023722 11851 raft_consensus.cc:2454] T cbdf3df0e269498c8bb57bb9badf204f P e6759c961f724ee1ab5dd4bc22b3411b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c5ef5580b16496bad3459d19814ccd7 in term 0.
I20260924 03:21:20.023834 11506 leader_election.cc:302] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c5ef5580b16496bad3459d19814ccd7, fcd549f721904765b980cba6e5fd4046; no voters:
I20260924 03:21:20.024292 11925 raft_consensus.cc:2781] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:21:20.024415 11925 raft_consensus.cc:482] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:21:20.024510 11925 raft_consensus.cc:3037] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:20.026566 11925 raft_consensus.cc:504] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:20.027187 11925 leader_election.cc:288] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7 [CANDIDATE]: Term 1 election: Requested vote from peers fcd549f721904765b980cba6e5fd4046 (127.10.12.130:36641), e6759c961f724ee1ab5dd4bc22b3411b (127.10.12.131:40969)
I20260924 03:21:20.027747 11851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf3df0e269498c8bb57bb9badf204f" candidate_uuid: "1c5ef5580b16496bad3459d19814ccd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6759c961f724ee1ab5dd4bc22b3411b"
I20260924 03:21:20.027786 11717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbdf3df0e269498c8bb57bb9badf204f" candidate_uuid: "1c5ef5580b16496bad3459d19814ccd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcd549f721904765b980cba6e5fd4046"
I20260924 03:21:20.028010 11851 raft_consensus.cc:3037] T cbdf3df0e269498c8bb57bb9badf204f P e6759c961f724ee1ab5dd4bc22b3411b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:20.028023 11717 raft_consensus.cc:3037] T cbdf3df0e269498c8bb57bb9badf204f P fcd549f721904765b980cba6e5fd4046 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:20.030886 11717 raft_consensus.cc:2454] T cbdf3df0e269498c8bb57bb9badf204f P fcd549f721904765b980cba6e5fd4046 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c5ef5580b16496bad3459d19814ccd7 in term 1.
I20260924 03:21:20.030871 11851 raft_consensus.cc:2454] T cbdf3df0e269498c8bb57bb9badf204f P e6759c961f724ee1ab5dd4bc22b3411b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c5ef5580b16496bad3459d19814ccd7 in term 1.
I20260924 03:21:20.031347 11506 leader_election.cc:302] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1c5ef5580b16496bad3459d19814ccd7, e6759c961f724ee1ab5dd4bc22b3411b, fcd549f721904765b980cba6e5fd4046; no voters:
I20260924 03:21:20.031680 11925 raft_consensus.cc:2781] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:20.032182 11925 raft_consensus.cc:686] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7 [term 1 LEADER]: Becoming Leader. State: Replica: 1c5ef5580b16496bad3459d19814ccd7, State: Running, Role: LEADER
I20260924 03:21:20.032603 11925 consensus_queue.cc:237] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } }
I20260924 03:21:20.035135 11966 tablet_replica.cc:439] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7: TxnStatusTablet state changed. Reason: New leader 1c5ef5580b16496bad3459d19814ccd7. Latest consensus state: current_term: 1 leader_uuid: "1c5ef5580b16496bad3459d19814ccd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } } }
I20260924 03:21:20.035591 11966 tablet_replica.cc:442] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:21:20.036530 11968 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 03:21:20.036715 11968 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 03:21:20.036908 11414 catalog_manager.cc:5495] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c5ef5580b16496bad3459d19814ccd7 (127.10.12.129). New cstate: current_term: 1 leader_uuid: "1c5ef5580b16496bad3459d19814ccd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } health_report { overall_health: UNKNOWN } } }
I20260924 03:21:20.100870 11917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:20.094746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54574 (local address 127.10.12.129:44489)
0924 03:21:20.095067 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:20.095074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:20.095398 (+ 324us) server_negotiation.cc:407] Connection header received
0924 03:21:20.095673 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:20.095678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:20.095780 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:20.095898 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:20.096292 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.097084 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.098409 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.098900 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.099011 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:20.099504 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:20.100191 (+ 687us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:20.100336 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:20.100449 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 03:21:20.100537 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:21:20.100858 11958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:20.095013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:44489 (local address 127.0.0.1:54574)
0924 03:21:20.095297 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:20.095314 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:20.095469 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:20.095944 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:20.095975 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:20.095989 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:21:20.096158 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:20.096167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.097217 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.097223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:20.098246 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:20.098258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.099065 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.099071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:20.099247 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:20.099314 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:21:20.100307 (+ 993us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:21:20.100332 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 03:21:20.100417 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 03:21:20.100582 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 03:21:20.102243 11927 consensus_queue.cc:1035] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:21:20.106832 11850 raft_consensus.cc:1261] T cbdf3df0e269498c8bb57bb9badf204f P e6759c961f724ee1ab5dd4bc22b3411b [term 1 FOLLOWER]: Refusing update from remote peer 1c5ef5580b16496bad3459d19814ccd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:21:20.107760 11717 raft_consensus.cc:1261] T cbdf3df0e269498c8bb57bb9badf204f P fcd549f721904765b980cba6e5fd4046 [term 1 FOLLOWER]: Refusing update from remote peer 1c5ef5580b16496bad3459d19814ccd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:21:20.107806 11966 consensus_queue.cc:1035] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:21:20.109460 11966 consensus_queue.cc:1035] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:21:20.145303 11938 tablet_replica.cc:439] T cbdf3df0e269498c8bb57bb9badf204f P e6759c961f724ee1ab5dd4bc22b3411b: TxnStatusTablet state changed. Reason: New leader 1c5ef5580b16496bad3459d19814ccd7. Latest consensus state: current_term: 1 leader_uuid: "1c5ef5580b16496bad3459d19814ccd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } } }
I20260924 03:21:20.145605 11938 tablet_replica.cc:442] T cbdf3df0e269498c8bb57bb9badf204f P e6759c961f724ee1ab5dd4bc22b3411b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:21:20.145874 11979 consensus_queue.cc:1035] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:21:20.146679 11951 tablet_replica.cc:439] T cbdf3df0e269498c8bb57bb9badf204f P fcd549f721904765b980cba6e5fd4046: TxnStatusTablet state changed. Reason: New leader 1c5ef5580b16496bad3459d19814ccd7. Latest consensus state: current_term: 1 leader_uuid: "1c5ef5580b16496bad3459d19814ccd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } } }
I20260924 03:21:20.147023 11951 tablet_replica.cc:442] T cbdf3df0e269498c8bb57bb9badf204f P fcd549f721904765b980cba6e5fd4046: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:21:20.150790 11971 mvcc.cc:205] Tried to move back new op lower bound from 7332741448105103360 to 7332741447817895936. Current Snapshot: MvccSnapshot[applied={T|T < 7332741448105103360}]
I20260924 03:21:20.155481 11973 mvcc.cc:205] Tried to move back new op lower bound from 7332741448105103360 to 7332741447817895936. Current Snapshot: MvccSnapshot[applied={T|T < 7332741448105103360}]
I20260924 03:21:20.161949 11938 tablet_replica.cc:439] T cbdf3df0e269498c8bb57bb9badf204f P e6759c961f724ee1ab5dd4bc22b3411b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c5ef5580b16496bad3459d19814ccd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } } }
I20260924 03:21:20.162248 11938 tablet_replica.cc:442] T cbdf3df0e269498c8bb57bb9badf204f P e6759c961f724ee1ab5dd4bc22b3411b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:21:20.163353 11979 tablet_replica.cc:439] T cbdf3df0e269498c8bb57bb9badf204f P fcd549f721904765b980cba6e5fd4046: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c5ef5580b16496bad3459d19814ccd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } } }
I20260924 03:21:20.163635 11979 tablet_replica.cc:442] T cbdf3df0e269498c8bb57bb9badf204f P fcd549f721904765b980cba6e5fd4046: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:21:20.176100 11966 tablet_replica.cc:439] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c5ef5580b16496bad3459d19814ccd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } } }
I20260924 03:21:20.176474 11966 tablet_replica.cc:442] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:21:20.183161 11966 tablet_replica.cc:439] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c5ef5580b16496bad3459d19814ccd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5ef5580b16496bad3459d19814ccd7" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 44489 } } peers { permanent_uuid: "fcd549f721904765b980cba6e5fd4046" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 36641 } } peers { permanent_uuid: "e6759c961f724ee1ab5dd4bc22b3411b" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40969 } } }
I20260924 03:21:20.183492 11966 tablet_replica.cc:442] T cbdf3df0e269498c8bb57bb9badf204f P 1c5ef5580b16496bad3459d19814ccd7: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:21:20.204808 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:20.196249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36626 (local address 127.10.12.130:36641)
0924 03:21:20.196435 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:20.196442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:20.196979 (+ 537us) server_negotiation.cc:407] Connection header received
0924 03:21:20.197363 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:20.197367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:20.197453 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:20.197595 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:20.198112 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.199364 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.201732 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.202170 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.202285 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:20.203146 (+ 861us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:20.203824 (+ 678us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:20.203980 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:20.204405 (+ 425us) server_negotiation.cc:299] Negotiation successful
0924 03:21:20.204512 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 03:21:20.226192 11993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:20.218350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:44489 (local address 127.10.12.130:48919)
0924 03:21:20.219020 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:20.219038 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:20.219178 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:20.219616 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:20.219619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:20.219631 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:21:20.219905 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:20.219913 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.221479 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.221484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:20.223387 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:20.223407 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.224862 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.224868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:20.225005 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:20.225085 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:21:20.225629 (+ 544us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:21:20.225640 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 03:21:20.225732 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 03:21:20.225862 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":320,"threads_started":1}
I20260924 03:21:20.226202 11917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:20.218475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:48919 (local address 127.10.12.129:44489)
0924 03:21:20.218737 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:20.218744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:20.219126 (+ 382us) server_negotiation.cc:407] Connection header received
0924 03:21:20.219303 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:20.219308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:20.219399 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:20.219527 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:20.220112 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.221251 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.223602 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.224622 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.224786 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:20.225326 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:20.225476 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:20.225572 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:20.225781 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 03:21:20.225875 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 03:21:20.283759 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:20.269654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:37919 (local address 127.10.12.130:36641)
0924 03:21:20.270396 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:20.270403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:20.276721 (+ 6318us) server_negotiation.cc:407] Connection header received
0924 03:21:20.276936 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:20.276941 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:20.277032 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:20.277112 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:20.278178 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.279240 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.281869 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.282351 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.282472 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:20.282994 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:20.283118 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:20.283196 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:20.283392 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 03:21:20.283493 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558}
I20260924 03:21:20.285058 11995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:20.269743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:36641 (local address 127.10.12.129:37919)
0924 03:21:20.276578 (+ 6835us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:20.276597 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:20.276781 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:20.277155 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:20.277160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:20.277173 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:21:20.277928 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:20.277951 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.279419 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.279425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:20.281656 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:20.281676 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.282570 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.282576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:20.282714 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:20.282786 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:21:20.283226 (+ 440us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:21:20.283240 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 03:21:20.284224 (+ 984us) client_negotiation.cc:240] Negotiation successful
0924 03:21:20.284501 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6615,"thread_start_us":6417,"threads_started":1}
I20260924 03:21:20.330848 11415 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:21:20.331138 11415 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:21:20.335084 11415 catalog_manager.cc:5782] T 00000000000000000000000000000000 P add047dafdab4e0fbbedef99257fca6b: Sending DeleteTablet for 3 replicas of tablet 72dbc4568e98414b82bc62957a1b00d8
I20260924 03:21:20.336189 11692 tablet_service.cc:1514] Processing DeleteTablet for tablet 72dbc4568e98414b82bc62957a1b00d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:21:20 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:39752
I20260924 03:21:20.336680 11831 tablet_service.cc:1514] Processing DeleteTablet for tablet 72dbc4568e98414b82bc62957a1b00d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:21:20 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:55572
I20260924 03:21:20.336951 11998 tablet_replica.cc:330] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046: stopping tablet replica
I20260924 03:21:20.337213 11998 raft_consensus.cc:2229] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:21:20.337889 11998 raft_consensus.cc:2258] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:21:20.339646 11998 ts_tablet_manager.cc:1905] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:21:20.347841 11998 ts_tablet_manager.cc:1918] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:21:20.348181 11998 log.cc:1196] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/72dbc4568e98414b82bc62957a1b00d8
I20260924 03:21:20.348806 11998 ts_tablet_manager.cc:1939] T 72dbc4568e98414b82bc62957a1b00d8 P fcd549f721904765b980cba6e5fd4046: Deleting consensus metadata
I20260924 03:21:20.350227 11403 catalog_manager.cc:4841] TS fcd549f721904765b980cba6e5fd4046 (127.10.12.130:36641): tablet 72dbc4568e98414b82bc62957a1b00d8 (table test-table [id=a21b79e8b85345479c23fee3a9492a43]) successfully deleted
I20260924 03:21:20.351408 11553 tablet_service.cc:1514] Processing DeleteTablet for tablet 72dbc4568e98414b82bc62957a1b00d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:21:20 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:55886
I20260924 03:21:20.352232 12001 tablet_replica.cc:330] T 72dbc4568e98414b82bc62957a1b00d8 P 1c5ef5580b16496bad3459d19814ccd7: stopping tablet replica
I20260924 03:21:20.352504 12001 raft_consensus.cc:2229] T 72dbc4568e98414b82bc62957a1b00d8 P 1c5ef5580b16496bad3459d19814ccd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:21:20.352910 12001 raft_consensus.cc:2258] T 72dbc4568e98414b82bc62957a1b00d8 P 1c5ef5580b16496bad3459d19814ccd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:21:20.354101 12001 ts_tablet_manager.cc:1905] T 72dbc4568e98414b82bc62957a1b00d8 P 1c5ef5580b16496bad3459d19814ccd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:21:20.363078 11999 tablet_replica.cc:330] T 72dbc4568e98414b82bc62957a1b00d8 P e6759c961f724ee1ab5dd4bc22b3411b: stopping tablet replica
I20260924 03:21:20.363058 12001 ts_tablet_manager.cc:1918] T 72dbc4568e98414b82bc62957a1b00d8 P 1c5ef5580b16496bad3459d19814ccd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:21:20.363219 12001 log.cc:1196] T 72dbc4568e98414b82bc62957a1b00d8 P 1c5ef5580b16496bad3459d19814ccd7: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-0/wal/wals/72dbc4568e98414b82bc62957a1b00d8
I20260924 03:21:20.363489 11999 raft_consensus.cc:2229] T 72dbc4568e98414b82bc62957a1b00d8 P e6759c961f724ee1ab5dd4bc22b3411b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:21:20.363757 11999 raft_consensus.cc:2258] T 72dbc4568e98414b82bc62957a1b00d8 P e6759c961f724ee1ab5dd4bc22b3411b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:21:20.363845 12001 ts_tablet_manager.cc:1939] T 72dbc4568e98414b82bc62957a1b00d8 P 1c5ef5580b16496bad3459d19814ccd7: Deleting consensus metadata
I20260924 03:21:20.364966 11999 ts_tablet_manager.cc:1905] T 72dbc4568e98414b82bc62957a1b00d8 P e6759c961f724ee1ab5dd4bc22b3411b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:21:20.365231 11400 catalog_manager.cc:4841] TS 1c5ef5580b16496bad3459d19814ccd7 (127.10.12.129:44489): tablet 72dbc4568e98414b82bc62957a1b00d8 (table test-table [id=a21b79e8b85345479c23fee3a9492a43]) successfully deleted
I20260924 03:21:20.372468 11999 ts_tablet_manager.cc:1918] T 72dbc4568e98414b82bc62957a1b00d8 P e6759c961f724ee1ab5dd4bc22b3411b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:21:20.372732 11999 log.cc:1196] T 72dbc4568e98414b82bc62957a1b00d8 P e6759c961f724ee1ab5dd4bc22b3411b: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/72dbc4568e98414b82bc62957a1b00d8
I20260924 03:21:20.373215 11999 ts_tablet_manager.cc:1939] T 72dbc4568e98414b82bc62957a1b00d8 P e6759c961f724ee1ab5dd4bc22b3411b: Deleting consensus metadata
I20260924 03:21:20.374532 11400 catalog_manager.cc:4841] TS e6759c961f724ee1ab5dd4bc22b3411b (127.10.12.131:40969): tablet 72dbc4568e98414b82bc62957a1b00d8 (table test-table [id=a21b79e8b85345479c23fee3a9492a43]) successfully deleted
Sep 24 03:21:20 dist-test-slave-00qs krb5kdc[11371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220079, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.12.129@KRBTEST.COM
I20260924 03:21:20.392820 11917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:20.362110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54582 (local address 127.10.12.129:44489)
0924 03:21:20.362334 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:20.362341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:20.369114 (+ 6773us) server_negotiation.cc:407] Connection header received
0924 03:21:20.369423 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:20.369429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:20.369524 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:20.369601 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:20.372234 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.373486 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.375729 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.376404 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.376537 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:20.382886 (+ 6349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:20.382906 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:20.382927 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:20.382993 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:20.388092 (+ 5099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:20.389207 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:20.389219 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:20.389223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:20.389437 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:20.390587 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:20.390593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:20.390597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:20.391179 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:20.391571 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:20.391871 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 03:21:20.392183 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260924 03:21:20.393651 11618 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54582
I20260924 03:21:20.433770 12005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:20.397272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48790 (local address 127.10.12.190:43383)
0924 03:21:20.413062 (+ 15790us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:20.413069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:20.413099 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:21:20.413179 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:20.413185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:20.413274 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:20.413448 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:20.416107 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.417459 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.420178 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.420913 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.421063 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:20.424623 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:20.424630 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:20.424635 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:20.424697 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:20.429001 (+ 4304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:20.430101 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:20.430108 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:20.430111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:20.430296 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:20.430784 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:20.430789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:20.430793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:20.431200 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:20.431467 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:20.432962 (+ 1495us) server_negotiation.cc:299] Negotiation successful
0924 03:21:20.433351 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15617,"thread_start_us":16283,"threads_started":1}
W20260924 03:21:20.434650 11415 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48790
I20260924 03:21:20.437362 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 11487
I20260924 03:21:20.470191 11917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:20.440821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54584 (local address 127.10.12.129:44489)
0924 03:21:20.441059 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:20.441068 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:20.449766 (+ 8698us) server_negotiation.cc:407] Connection header received
0924 03:21:20.450695 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:20.450704 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:20.450826 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:20.450919 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:20.453601 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.454814 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.456918 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.457658 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.457833 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:20.461110 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:20.461119 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:20.461124 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:20.461179 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:20.466263 (+ 5084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:20.467509 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:20.467515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:20.467519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:20.467741 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:20.468327 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:20.468333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:20.468337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:20.468744 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:20.469034 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:20.469510 (+ 476us) server_negotiation.cc:299] Negotiation successful
0924 03:21:20.469797 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 03:21:20.470880 11618 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54584
W20260924 03:21:20.471627 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 11487: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:21:20.471844 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 11487
I20260924 03:21:20.500231 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 11627
Sep 24 03:21:20 dist-test-slave-00qs krb5kdc[11371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220079, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.12.130@KRBTEST.COM
I20260924 03:21:20.521347 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:20.501235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36642 (local address 127.10.12.130:36641)
0924 03:21:20.501464 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:20.501469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:20.501507 (+ 38us) server_negotiation.cc:407] Connection header received
0924 03:21:20.501781 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:20.501787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:20.501887 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:20.501996 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:20.503629 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.504612 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.506675 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.507331 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.507475 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:20.512917 (+ 5442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:20.512936 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:20.512953 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:20.512999 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:20.517775 (+ 4776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:20.518668 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:20.518675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:20.518678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:20.518838 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:20.519223 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:20.519226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:20.519228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:20.519708 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:20.520069 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:20.520727 (+ 658us) server_negotiation.cc:299] Negotiation successful
0924 03:21:20.521002 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260924 03:21:20.522042 11757 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36642
W20260924 03:21:20.522758 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 11627: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:21:20.522946 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 11627
I20260924 03:21:20.538074 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 11766
Sep 24 03:21:20 dist-test-slave-00qs krb5kdc[11371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220079, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.12.131@KRBTEST.COM
I20260924 03:21:20.555094 12007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:20.538965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41232 (local address 127.10.12.131:40969)
0924 03:21:20.539492 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:20.539496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:20.539513 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:21:20.539567 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:20.539571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:20.539642 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:20.539761 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:20.541196 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.541958 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.543273 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:20.543673 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:20.543778 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:20.548661 (+ 4883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:20.548678 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:20.548695 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:20.548776 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:20.552181 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:20.552990 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:20.552998 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:20.553001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:20.553131 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:20.553584 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:20.553588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:20.553591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:20.554007 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:20.554258 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:20.554601 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 03:21:20.554818 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":298,"threads_started":1}
W20260924 03:21:20.555766 11896 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41232
W20260924 03:21:20.556434 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 11766: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:21:20.556583 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 11766
I20260924 03:21:20.573735 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 11384
I20260924 03:21:20.575009 11456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:20.692106 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 11384
2026-09-24T03:21:20Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4927 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:21:20 dist-test-slave-00qs krb5kdc[12011](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:21:20 dist-test-slave-00qs krb5kdc[12011](info): set up 2 sockets
Sep 24 03:21:20 dist-test-slave-00qs krb5kdc[12011](info): commencing operation
krb5kdc: starting...
W20260924 03:21:21.716653 10290 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:21:21 dist-test-slave-00qs krb5kdc[12011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:21:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:21:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:21:21.912101 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:39047
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:44809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:39047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:22.257438 12024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:22.257771 12024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:22.257891 12024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:22.265501 12024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:21:22.265798 12024 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 03:21:22.265962 12024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:22.266055 12024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:22.266124 12024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:21:22.266209 12024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:21:22.276988 12024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:44809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:39047
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:39047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:22.278360 12024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:22.279776 12024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:22.292915 12032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:22.292862 12029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:22.293087 12030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:22.293639 12024 server_base.cc:1034] running on GCE node
I20260924 03:21:22.294528 12024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:22.296250 12024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:22.297504 12024 hybrid_clock.cc:648] HybridClock initialized: now 1790220082297438 us; error 47 us; skew 500 ppm
I20260924 03:21:22.297883 12024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:22 dist-test-slave-00qs krb5kdc[12011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220082, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:22.301659 12024 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:21:22.303501 12024 webserver.cc:460] Webserver started at http://127.10.12.190:35215/ using document root <none> and password file <none>
I20260924 03:21:22.304392 12024 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:22.304503 12024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:22.304837 12024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:22.308198 12024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "63f05b4c617846e9970318185e930366"
format_stamp: "Formatted at 2026-09-24 03:21:22 on dist-test-slave-00qs"
server_key: "f1371c2d5c9cdccd787ae6530e09ed8b"
server_key_iv: "e0f72f1008178694c87d22df8911248f"
server_key_version: "encryptionkey@0"
I20260924 03:21:22.309098 12024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "63f05b4c617846e9970318185e930366"
format_stamp: "Formatted at 2026-09-24 03:21:22 on dist-test-slave-00qs"
server_key: "f1371c2d5c9cdccd787ae6530e09ed8b"
server_key_iv: "e0f72f1008178694c87d22df8911248f"
server_key_version: "encryptionkey@0"
I20260924 03:21:22.315373 12024 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 03:21:22.319556 12039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:22.321545 12024 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260924 03:21:22.321758 12024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "63f05b4c617846e9970318185e930366"
format_stamp: "Formatted at 2026-09-24 03:21:22 on dist-test-slave-00qs"
server_key: "f1371c2d5c9cdccd787ae6530e09ed8b"
server_key_iv: "e0f72f1008178694c87d22df8911248f"
server_key_version: "encryptionkey@0"
I20260924 03:21:22.321990 12024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:22.339880 12024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:22.345116 12024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:22.345491 12024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:22.368587 12024 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:39047
I20260924 03:21:22.368621 12100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:39047 every 8 connection(s)
I20260924 03:21:22.370018 12024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:21:22.371891 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 12024
I20260924 03:21:22.372229 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:21:22.373122 10290 external_mini_cluster.cc:1428] Setting key db1d360776b6f6e75250cc792423c7a1
I20260924 03:21:22.375732 12101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 03:21:22 dist-test-slave-00qs krb5kdc[12011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:22.390033 12101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366: Bootstrap starting.
I20260924 03:21:22.393903 12101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:22.395100 12101 log.cc:824] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:22.398695 12101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366: No bootstrap required, opened a new log
I20260924 03:21:22.399741 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:22.375470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43486 (local address 127.10.12.190:39047)
0924 03:21:22.376400 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:22.376413 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:22.376462 (+ 49us) server_negotiation.cc:407] Connection header received
0924 03:21:22.378219 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:22.378255 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:22.378796 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:22.379190 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:22.381377 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:22.382808 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:22.384640 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:22.385443 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:22.385596 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:22.390455 (+ 4859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:22.390465 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:22.390483 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:22.390533 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:22.394683 (+ 4150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:22.395673 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:22.395685 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:22.395707 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:22.396054 (+ 347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:22.396576 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:22.396581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:22.396584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:22.397162 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:22.397387 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:22.398073 (+ 686us) server_negotiation.cc:299] Negotiation successful
0924 03:21:22.398563 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":629,"thread_start_us":414,"threads_started":1}
I20260924 03:21:22.407846 12101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f05b4c617846e9970318185e930366" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39047 } }
I20260924 03:21:22.408354 12101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:22.408460 12101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63f05b4c617846e9970318185e930366, State: Initialized, Role: FOLLOWER
I20260924 03:21:22.409078 12101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f05b4c617846e9970318185e930366" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39047 } }
I20260924 03:21:22.409323 12101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:21:22.409459 12101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:21:22.409623 12101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:22.412599 12101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f05b4c617846e9970318185e930366" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39047 } }
I20260924 03:21:22.413240 12101 leader_election.cc:302] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63f05b4c617846e9970318185e930366; no voters:
I20260924 03:21:22.413929 12101 leader_election.cc:288] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:21:22.414067 12106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:22.415537 12106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366 [term 1 LEADER]: Becoming Leader. State: Replica: 63f05b4c617846e9970318185e930366, State: Running, Role: LEADER
I20260924 03:21:22.416191 12106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f05b4c617846e9970318185e930366" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39047 } }
I20260924 03:21:22.416801 12101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:21:22.420363 12107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63f05b4c617846e9970318185e930366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f05b4c617846e9970318185e930366" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39047 } } }
I20260924 03:21:22.420043 12108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63f05b4c617846e9970318185e930366. Latest consensus state: current_term: 1 leader_uuid: "63f05b4c617846e9970318185e930366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f05b4c617846e9970318185e930366" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39047 } } }
I20260924 03:21:22.420737 12108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:22.421465 12107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:22.422690 12114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:21:22.428592 12114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:21:22.440255 12114 catalog_manager.cc:1347] Generated new cluster ID: 94b9f274a97c429295d6257b5f636be4
I20260924 03:21:22.440421 12114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:21:22.456971 12114 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:21:22.458808 12114 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:21:22.466198 12114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366: Generated new TSK 0
I20260924 03:21:22.468103 12114 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
I20260924 03:21:22.518437 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:39047
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:44809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:22.868755 12127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:22.869082 12127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:22.869227 12127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:22.877719 12127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:22.877876 12127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:22.878008 12127 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 03:21:22.878079 12127 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:21:22.878189 12127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:21:22.888968 12127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:44809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.12.190:39047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:22.890385 12127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:22.891845 12127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:22.907255 12135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:22.907442 12133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:22.908242 12132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:22.908905 12127 server_base.cc:1034] running on GCE node
I20260924 03:21:22.910071 12127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:22.911688 12127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:22.912941 12127 hybrid_clock.cc:648] HybridClock initialized: now 1790220082912855 us; error 77 us; skew 500 ppm
I20260924 03:21:22.913302 12127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:22 dist-test-slave-00qs krb5kdc[12011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220082, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:22.916807 12127 init.cc:377] Logged in from keytab as kudu/127.10.12.129@KRBTEST.COM (short username kudu)
I20260924 03:21:22.918752 12127 webserver.cc:460] Webserver started at http://127.10.12.129:36491/ using document root <none> and password file <none>
I20260924 03:21:22.919543 12127 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:22.919682 12127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:22.920063 12127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:22.923511 12127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "8188e2acd6ea4edb866fb2c07959d533"
format_stamp: "Formatted at 2026-09-24 03:21:22 on dist-test-slave-00qs"
server_key: "2bacfa24f5f3d1f51ed213c83c42cabc"
server_key_iv: "7ae2e8309f9a3d10dc87542d8a8fd6da"
server_key_version: "encryptionkey@0"
I20260924 03:21:22.924403 12127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "8188e2acd6ea4edb866fb2c07959d533"
format_stamp: "Formatted at 2026-09-24 03:21:22 on dist-test-slave-00qs"
server_key: "2bacfa24f5f3d1f51ed213c83c42cabc"
server_key_iv: "7ae2e8309f9a3d10dc87542d8a8fd6da"
server_key_version: "encryptionkey@0"
I20260924 03:21:22.931640 12127 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s
I20260924 03:21:22.937069 12142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:22.938802 12127 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:21:22.938997 12127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "8188e2acd6ea4edb866fb2c07959d533"
format_stamp: "Formatted at 2026-09-24 03:21:22 on dist-test-slave-00qs"
server_key: "2bacfa24f5f3d1f51ed213c83c42cabc"
server_key_iv: "7ae2e8309f9a3d10dc87542d8a8fd6da"
server_key_version: "encryptionkey@0"
I20260924 03:21:22.939227 12127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:22.961511 12127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:22.966503 12127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:22.966801 12127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:22.970422 12127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:22.970542 12127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:22.970691 12127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:22.970768 12127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:21:22 dist-test-slave-00qs krb5kdc[12011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220082, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:22.996817 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:22.971356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:39359 (local address 127.10.12.190:39047)
0924 03:21:22.971837 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:22.971844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:22.973300 (+ 1456us) server_negotiation.cc:407] Connection header received
0924 03:21:22.976365 (+ 3065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:22.976372 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:22.976474 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:22.976611 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:22.979508 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:22.980529 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:22.983768 (+ 3239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:22.984345 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:22.984498 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:22.989822 (+ 5324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:22.989828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:22.989832 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:22.989880 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:22.993165 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:22.994310 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:22.994316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:22.994321 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:22.994464 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:22.994945 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:22.994949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:22.994951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:22.995233 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:22.995370 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:22.996198 (+ 828us) server_negotiation.cc:299] Negotiation successful
0924 03:21:22.996452 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":226,"threads_started":1}
I20260924 03:21:22.998383 12152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:22.972161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39047 (local address 127.10.12.129:39359)
0924 03:21:22.973071 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:22.973138 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:22.975906 (+ 2768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:22.976977 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:22.976997 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:22.978002 (+ 1005us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:22.979221 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:22.979262 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:22.980791 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:22.980797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:22.983541 (+ 2744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:22.983565 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:22.984682 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:22.984691 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:22.984961 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:22.985868 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:22.985913 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:22.989511 (+ 3598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:22.993416 (+ 3905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:22.993432 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:22.993460 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:22.994164 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:22.994594 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:22.994598 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:22.994601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:22.994828 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:22.995384 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:22.995394 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:22.995786 (+ 392us) client_negotiation.cc:769] Sending connection context
0924 03:21:22.996102 (+ 316us) client_negotiation.cc:240] Negotiation successful
0924 03:21:22.996734 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":321,"threads_started":1}
I20260924 03:21:23.014570 12127 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:38037
I20260924 03:21:23.014591 12262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:38037 every 8 connection(s)
I20260924 03:21:23.016276 12127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:21:23.019374 12263 heartbeater.cc:347] Connected to a master server at 127.10.12.190:39047
I20260924 03:21:23.019690 12263 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:23.020506 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 12127
I20260924 03:21:23.020699 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:21:23.020750 12263 heartbeater.cc:511] Master 127.10.12.190:39047 requested a full tablet report, sending...
I20260924 03:21:23.021523 10290 external_mini_cluster.cc:1428] Setting key 0186d00edfd9fbdf34f839e21668e096
I20260924 03:21:23.023183 12056 ts_manager.cc:194] Registered new tserver with Master: 8188e2acd6ea4edb866fb2c07959d533 (127.10.12.129:38037)
I20260924 03:21:23.025123 12056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.129@KRBTEST.COM'} at 127.10.12.129:39359
I20260924 03:21:23.027717 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:23.006665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:53241 (local address 127.10.12.190:39047)
0924 03:21:23.007056 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:23.007066 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:23.012823 (+ 5757us) server_negotiation.cc:407] Connection header received
0924 03:21:23.013011 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:23.013017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:23.013096 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:23.013182 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:23.014479 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:23.015263 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:23.017350 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:23.018015 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:23.018132 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:23.021074 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:23.021082 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:23.021088 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:23.021138 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:23.024429 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:23.025335 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:23.025340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:23.025353 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:23.025531 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:23.025941 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:23.025946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:23.025950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:23.026321 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:23.026482 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:23.027146 (+ 664us) server_negotiation.cc:299] Negotiation successful
0924 03:21:23.027401 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260924 03:21:23.027858 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:23.006671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39047 (local address 127.10.12.129:53241)
0924 03:21:23.012652 (+ 5981us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:23.012675 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:23.012883 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:23.013242 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:23.013247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:23.013917 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:23.014286 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:23.014303 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:23.015411 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:23.015416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:23.016934 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:23.016977 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:23.018475 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:23.018481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:23.018651 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:23.019542 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:23.019564 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:23.020741 (+ 1177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:23.024591 (+ 3850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:23.024597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:23.024602 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:23.025158 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:23.025657 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:23.025662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:23.025666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:23.025818 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:23.026541 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:23.026546 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:23.026755 (+ 209us) client_negotiation.cc:769] Sending connection context
0924 03:21:23.027010 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 03:21:23.027403 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5821,"thread_start_us":5692,"threads_started":1}
WARNING: no policy specified for kudu/127.10.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
I20260924 03:21:23.069334 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:39047
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:44809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:23.406406 12267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:23.406729 12267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:23.406838 12267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:23.415247 12267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:23.415403 12267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:23.415499 12267 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 03:21:23.415560 12267 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:21:23.415642 12267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:21:23.426401 12267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:44809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.12.190:39047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:23.427709 12267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:23.429050 12267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:23.443732 12273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:23.444118 12272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:23.444186 12275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:23.445012 12267 server_base.cc:1034] running on GCE node
I20260924 03:21:23.445858 12267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:23.447077 12267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:23.448493 12267 hybrid_clock.cc:648] HybridClock initialized: now 1790220083448385 us; error 76 us; skew 500 ppm
I20260924 03:21:23.449107 12267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:23 dist-test-slave-00qs krb5kdc[12011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220083, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:23.453153 12267 init.cc:377] Logged in from keytab as kudu/127.10.12.130@KRBTEST.COM (short username kudu)
I20260924 03:21:23.455363 12267 webserver.cc:460] Webserver started at http://127.10.12.130:44459/ using document root <none> and password file <none>
I20260924 03:21:23.456202 12267 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:23.456343 12267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:23.456667 12267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:23.459867 12267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "61d88c04b6344d54a3aadac52a4b4ca1"
format_stamp: "Formatted at 2026-09-24 03:21:23 on dist-test-slave-00qs"
server_key: "5127bee4dac28767016a5ed9e65ffa5a"
server_key_iv: "21141bf28f64748e4f4b71317aef7ff2"
server_key_version: "encryptionkey@0"
I20260924 03:21:23.460736 12267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "61d88c04b6344d54a3aadac52a4b4ca1"
format_stamp: "Formatted at 2026-09-24 03:21:23 on dist-test-slave-00qs"
server_key: "5127bee4dac28767016a5ed9e65ffa5a"
server_key_iv: "21141bf28f64748e4f4b71317aef7ff2"
server_key_version: "encryptionkey@0"
I20260924 03:21:23.466535 12267 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 03:21:23.471073 12282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:23.473119 12267 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 03:21:23.473333 12267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "61d88c04b6344d54a3aadac52a4b4ca1"
format_stamp: "Formatted at 2026-09-24 03:21:23 on dist-test-slave-00qs"
server_key: "5127bee4dac28767016a5ed9e65ffa5a"
server_key_iv: "21141bf28f64748e4f4b71317aef7ff2"
server_key_version: "encryptionkey@0"
I20260924 03:21:23.473590 12267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:23.499166 12267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:23.504567 12267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:23.504885 12267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:23.508100 12267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:23.508216 12267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:23.508335 12267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:23.508411 12267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:21:23 dist-test-slave-00qs krb5kdc[12011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220083, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:23.536743 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:23.509115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:55909 (local address 127.10.12.190:39047)
0924 03:21:23.509302 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:23.509309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:23.511081 (+ 1772us) server_negotiation.cc:407] Connection header received
0924 03:21:23.513548 (+ 2467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:23.513554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:23.513649 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:23.513738 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:23.515469 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:23.516577 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:23.520377 (+ 3800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:23.520942 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:23.521111 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:23.527626 (+ 6515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:23.527635 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:23.527640 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:23.527695 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:23.530636 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:23.532400 (+ 1764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:23.532408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:23.532411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:23.532615 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:23.533266 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:23.533272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:23.533276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:23.533651 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:23.533814 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:23.536033 (+ 2219us) server_negotiation.cc:299] Negotiation successful
0924 03:21:23.536359 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 03:21:23.538568 12291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:23.509854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39047 (local address 127.10.12.130:55909)
0924 03:21:23.510867 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:23.510914 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:23.513162 (+ 2248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:23.514038 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:23.514047 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:23.514625 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:23.515258 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:23.515278 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:23.516990 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:23.516997 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:23.520106 (+ 3109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:23.520140 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:23.521274 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:23.521283 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:23.521642 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:23.522726 (+ 1084us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:23.522764 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:23.527186 (+ 4422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:23.531010 (+ 3824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:23.531025 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:23.531053 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:23.531911 (+ 858us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:23.532839 (+ 928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:23.532843 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:23.532846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:23.533100 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:23.534014 (+ 914us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:23.534024 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:23.535220 (+ 1196us) client_negotiation.cc:769] Sending connection context
0924 03:21:23.535816 (+ 596us) client_negotiation.cc:240] Negotiation successful
0924 03:21:23.537012 (+ 1196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":395,"threads_started":1}
I20260924 03:21:23.562139 12267 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:42369
I20260924 03:21:23.562209 12401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:42369 every 8 connection(s)
I20260924 03:21:23.563486 12267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:21:23.564255 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:23.547225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:53675 (local address 127.10.12.190:39047)
0924 03:21:23.547445 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:23.547452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:23.548356 (+ 904us) server_negotiation.cc:407] Connection header received
0924 03:21:23.548544 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:23.548550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:23.548640 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:23.548742 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:23.550398 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:23.551595 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:23.553989 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:23.554652 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:23.554819 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:23.557612 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:23.557620 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:23.557626 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:23.557675 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:23.560951 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:23.562026 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:23.562034 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:23.562037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:23.562231 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:23.562668 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:23.562672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:23.562674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:23.563061 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:23.563202 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:23.563644 (+ 442us) server_negotiation.cc:299] Negotiation successful
0924 03:21:23.563879 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 03:21:23.564394 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:23.547400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39047 (local address 127.10.12.130:53675)
0924 03:21:23.548229 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:23.548251 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:23.548416 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:23.548846 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:23.548853 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:23.549587 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:23.550131 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:23.550152 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:23.551862 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:23.551869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:23.553758 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:23.553778 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:23.554933 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:23.554944 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:23.555157 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:23.555927 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:23.555986 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:23.557344 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:23.561160 (+ 3816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:23.561167 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:23.561172 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:23.561842 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:23.562402 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:23.562408 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:23.562416 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:23.562544 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:23.563184 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:23.563189 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:23.563380 (+ 191us) client_negotiation.cc:769] Sending connection context
0924 03:21:23.563513 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 03:21:23.563887 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":472,"threads_started":1}
I20260924 03:21:23.566396 12402 heartbeater.cc:347] Connected to a master server at 127.10.12.190:39047
I20260924 03:21:23.566623 12402 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:23.567293 12402 heartbeater.cc:511] Master 127.10.12.190:39047 requested a full tablet report, sending...
I20260924 03:21:23.568662 12056 ts_manager.cc:194] Registered new tserver with Master: 61d88c04b6344d54a3aadac52a4b4ca1 (127.10.12.130:42369)
I20260924 03:21:23.569325 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 12267
I20260924 03:21:23.569494 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:21:23.569730 12056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.130@KRBTEST.COM'} at 127.10.12.130:55909
I20260924 03:21:23.570021 10290 external_mini_cluster.cc:1428] Setting key 7b0d94cef0e8ad4d2b4074f3cc75d070
WARNING: no policy specified for kudu/127.10.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
I20260924 03:21:23.616425 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:39047
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:44809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:23.969252 12406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:23.969611 12406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:23.969743 12406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:23.977595 12406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:23.977757 12406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:23.977908 12406 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 03:21:23.977979 12406 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:21:23.978085 12406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:21:23.990563 12406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:44809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.12.190:39047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:23.991945 12406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:23.993323 12406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:24.007144 12414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:24.007253 12411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:24.007256 12412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:24.008132 12406 server_base.cc:1034] running on GCE node
I20260924 03:21:24.008752 12406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:24.009750 12406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:24.010967 12406 hybrid_clock.cc:648] HybridClock initialized: now 1790220084010916 us; error 43 us; skew 500 ppm
I20260924 03:21:24.011310 12406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:24 dist-test-slave-00qs krb5kdc[12011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220084, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:24.015110 12406 init.cc:377] Logged in from keytab as kudu/127.10.12.131@KRBTEST.COM (short username kudu)
I20260924 03:21:24.017115 12406 webserver.cc:460] Webserver started at http://127.10.12.131:36091/ using document root <none> and password file <none>
I20260924 03:21:24.017918 12406 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:24.018075 12406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:24.018404 12406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:24.021661 12406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "d0b7249d472b430ab4dfc3e312d4ea02"
format_stamp: "Formatted at 2026-09-24 03:21:24 on dist-test-slave-00qs"
server_key: "32d04fcf0cdfe4840ae4e9333cbea033"
server_key_iv: "8e0fa99b1ca33779c79b835734695022"
server_key_version: "encryptionkey@0"
I20260924 03:21:24.022568 12406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "d0b7249d472b430ab4dfc3e312d4ea02"
format_stamp: "Formatted at 2026-09-24 03:21:24 on dist-test-slave-00qs"
server_key: "32d04fcf0cdfe4840ae4e9333cbea033"
server_key_iv: "8e0fa99b1ca33779c79b835734695022"
server_key_version: "encryptionkey@0"
I20260924 03:21:24.029449 12406 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260924 03:21:24.029681 12263 heartbeater.cc:503] Master 127.10.12.190:39047 was elected leader, sending a full tablet report...
I20260924 03:21:24.034531 12421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:24.036458 12406 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260924 03:21:24.036659 12406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "d0b7249d472b430ab4dfc3e312d4ea02"
format_stamp: "Formatted at 2026-09-24 03:21:24 on dist-test-slave-00qs"
server_key: "32d04fcf0cdfe4840ae4e9333cbea033"
server_key_iv: "8e0fa99b1ca33779c79b835734695022"
server_key_version: "encryptionkey@0"
I20260924 03:21:24.036868 12406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:24.054796 12406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:24.059748 12406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:24.060070 12406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:24.062804 12406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:24.062940 12406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:24.063056 12406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:24.063128 12406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:21:24 dist-test-slave-00qs krb5kdc[12011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220084, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:24.089550 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:24.063690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:43853 (local address 127.10.12.190:39047)
0924 03:21:24.063941 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:24.063973 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:24.065750 (+ 1777us) server_negotiation.cc:407] Connection header received
0924 03:21:24.068635 (+ 2885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:24.068642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:24.068762 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:24.068870 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:24.071403 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.072496 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.076202 (+ 3706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.076751 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.076917 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:24.082393 (+ 5476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:24.082399 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:24.082403 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:24.082444 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:24.085534 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:24.086477 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:24.086483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:24.086486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:24.086628 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:24.087159 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:24.087164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:24.087167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:24.087566 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:24.087725 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:24.088915 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0924 03:21:24.089171 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260924 03:21:24.090809 12430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:24.064516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39047 (local address 127.10.12.131:43853)
0924 03:21:24.065530 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:24.065584 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:24.067981 (+ 2397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:24.069185 (+ 1204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:24.069195 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:24.070152 (+ 957us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:24.071168 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:24.071199 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.072717 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.072724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:24.075901 (+ 3177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:24.075932 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.077091 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.077101 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:24.077468 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:24.078557 (+ 1089us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:24.078596 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:24.082094 (+ 3498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:24.085731 (+ 3637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:24.085743 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:24.085760 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:24.086315 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:24.086746 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:24.086752 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:24.086755 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:24.086995 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:24.087812 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:24.087822 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:24.088444 (+ 622us) client_negotiation.cc:769] Sending connection context
0924 03:21:24.088735 (+ 291us) client_negotiation.cc:240] Negotiation successful
0924 03:21:24.089334 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":444,"threads_started":1}
I20260924 03:21:24.123971 12406 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:35321
I20260924 03:21:24.124243 12540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:35321 every 8 connection(s)
I20260924 03:21:24.124890 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:24.107416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39047 (local address 127.10.12.131:50637)
0924 03:21:24.108068 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:24.108086 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:24.108236 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:24.108693 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:24.108698 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:24.109178 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:24.109524 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:24.109534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.111158 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.111163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:24.112340 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:24.112358 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.113489 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.113498 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:24.113986 (+ 488us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:24.115367 (+ 1381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:24.115404 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:24.116824 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:24.121102 (+ 4278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:24.121108 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:24.121113 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:24.121732 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:24.122293 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:24.122298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:24.122302 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:24.122473 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:24.123399 (+ 926us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:24.123405 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:24.123662 (+ 257us) client_negotiation.cc:769] Sending connection context
0924 03:21:24.123823 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 03:21:24.124396 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":333,"threads_started":1}
I20260924 03:21:24.124886 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:24.107372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:50637 (local address 127.10.12.190:39047)
0924 03:21:24.107562 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:24.107567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:24.108172 (+ 605us) server_negotiation.cc:407] Connection header received
0924 03:21:24.108369 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:24.108376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:24.108490 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:24.108620 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:24.109678 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.110960 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.112531 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.113224 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.113364 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:24.117161 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:24.117169 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:24.117174 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:24.117224 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:24.120875 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:24.121926 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:24.121933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:24.121937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:24.122143 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:24.122614 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:24.122620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:24.122623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:24.123120 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:24.123324 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:24.124234 (+ 910us) server_negotiation.cc:299] Negotiation successful
0924 03:21:24.124500 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 03:21:24.126343 12406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:21:24.129690 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 12406
I20260924 03:21:24.129886 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:21:24.130507 10290 external_mini_cluster.cc:1428] Setting key 18fa65e526f5ceae20cec31916948a19
I20260924 03:21:24.131035 12541 heartbeater.cc:347] Connected to a master server at 127.10.12.190:39047
I20260924 03:21:24.131300 12541 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:24.132144 12541 heartbeater.cc:511] Master 127.10.12.190:39047 requested a full tablet report, sending...
I20260924 03:21:24.133616 12055 ts_manager.cc:194] Registered new tserver with Master: d0b7249d472b430ab4dfc3e312d4ea02 (127.10.12.131:35321)
I20260924 03:21:24.134526 12055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.131@KRBTEST.COM'} at 127.10.12.131:43853
I20260924 03:21:24.146842 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 03:21:24 dist-test-slave-00qs krb5kdc[12011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 03:21:24 dist-test-slave-00qs krb5kdc[12011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:24.199265 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:24.182105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43488 (local address 127.10.12.190:39047)
0924 03:21:24.182371 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:24.182378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:24.182528 (+ 150us) server_negotiation.cc:407] Connection header received
0924 03:21:24.182717 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:24.182722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:24.182787 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:24.182943 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:24.184456 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.185205 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.186520 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.186956 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.187076 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:24.192477 (+ 5401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:24.192484 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:24.192488 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:24.192537 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:24.196122 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:24.197023 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:24.197029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:24.197031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:24.197217 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:24.197664 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:24.197668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:24.197670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:24.198004 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:24.198253 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:24.198625 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 03:21:24.198885 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 03:21:24.204226 12055 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:21:24.211266 12055 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:21:24.249512 12556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:24.227125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44256 (local address 127.10.12.131:35321)
0924 03:21:24.238444 (+ 11319us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:24.238455 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:24.240348 (+ 1893us) server_negotiation.cc:407] Connection header received
0924 03:21:24.240498 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:24.240508 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:24.240818 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:24.241727 (+ 909us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:24.242726 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.244200 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.246879 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.248265 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.248409 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:24.248659 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:24.248977 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 03:21:24.249159 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11115,"thread_start_us":10898,"threads_started":1}
I20260924 03:21:24.249529 12557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:24.226664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:35321 (local address 127.0.0.1:44256)
0924 03:21:24.239094 (+ 12430us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:24.239154 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:24.239426 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:24.241934 (+ 2508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:24.241942 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:24.241979 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:24.242459 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:24.242472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.244388 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.244393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:24.246656 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:24.246697 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.248462 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.248469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:24.248670 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:24.248737 (+ 67us) client_negotiation.cc:769] Sending connection context
0924 03:21:24.248894 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 03:21:24.249100 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12119,"thread_start_us":11968,"threads_started":1}
I20260924 03:21:24.250769 12555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:24.227121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40830 (local address 127.10.12.130:42369)
0924 03:21:24.241865 (+ 14744us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:24.241874 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:24.241903 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:21:24.242010 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:24.242017 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:24.242317 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:24.242498 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:24.243025 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.244537 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.247514 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.248860 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.249001 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:24.249180 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:24.250325 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0924 03:21:24.250497 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14586,"thread_start_us":14441,"threads_started":1}
I20260924 03:21:24.250823 12558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:24.226804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:42369 (local address 127.0.0.1:40830)
0924 03:21:24.239541 (+ 12737us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:24.239561 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:24.239685 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:24.242517 (+ 2832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:24.242522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:24.242546 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:24.242875 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:24.242885 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.244719 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.244728 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:24.247319 (+ 2591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:24.247335 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.249336 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.249342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:24.249534 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:24.249580 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 03:21:24.250243 (+ 663us) client_negotiation.cc:240] Negotiation successful
0924 03:21:24.250394 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12602,"mutex_wait_us":114,"thread_start_us":15225,"threads_started":1}
I20260924 03:21:24.251387 12559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:24.230173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50114 (local address 127.10.12.129:38037)
0924 03:21:24.241395 (+ 11222us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:24.241407 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:24.242131 (+ 724us) server_negotiation.cc:407] Connection header received
0924 03:21:24.242315 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:24.242323 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:24.242547 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:24.242736 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:24.243561 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.245143 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.247766 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.249066 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.249192 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:24.249380 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:24.250933 (+ 1553us) server_negotiation.cc:299] Negotiation successful
0924 03:21:24.251112 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11008,"thread_start_us":10892,"threads_started":1}
I20260924 03:21:24.252050 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:24.229943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:38037 (local address 127.0.0.1:50114)
0924 03:21:24.241879 (+ 11936us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:24.241899 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:24.242011 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:24.242959 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:24.242963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:24.242990 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:24.243370 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:24.243384 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.245360 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.245369 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:24.247601 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:24.247611 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.249540 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.249547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:24.249743 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:24.249774 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 03:21:24.249850 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 03:21:24.251650 (+ 1800us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11756,"mutex_wait_us":40,"thread_start_us":10595,"threads_started":1}
I20260924 03:21:24.253818 12471 tablet_service.cc:1467] Processing CreateTablet for tablet 61cd6359c42a4603bd45df53d3acd75a (DEFAULT_TABLE table=test-table [id=dc7c4dbc46e04a98a647307af2f38c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:24.255276 12471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61cd6359c42a4603bd45df53d3acd75a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:24.256000 12193 tablet_service.cc:1467] Processing CreateTablet for tablet 61cd6359c42a4603bd45df53d3acd75a (DEFAULT_TABLE table=test-table [id=dc7c4dbc46e04a98a647307af2f38c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:24.257496 12193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61cd6359c42a4603bd45df53d3acd75a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:24.258098 12332 tablet_service.cc:1467] Processing CreateTablet for tablet 61cd6359c42a4603bd45df53d3acd75a (DEFAULT_TABLE table=test-table [id=dc7c4dbc46e04a98a647307af2f38c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:24.260035 12332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61cd6359c42a4603bd45df53d3acd75a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:24.268733 12560 tablet_bootstrap.cc:492] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02: Bootstrap starting.
I20260924 03:21:24.272183 12560 tablet_bootstrap.cc:654] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:24.272925 12561 tablet_bootstrap.cc:492] T 61cd6359c42a4603bd45df53d3acd75a P 8188e2acd6ea4edb866fb2c07959d533: Bootstrap starting.
I20260924 03:21:24.274329 12560 log.cc:824] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:24.276976 12563 tablet_bootstrap.cc:492] T 61cd6359c42a4603bd45df53d3acd75a P 61d88c04b6344d54a3aadac52a4b4ca1: Bootstrap starting.
I20260924 03:21:24.278630 12561 tablet_bootstrap.cc:654] T 61cd6359c42a4603bd45df53d3acd75a P 8188e2acd6ea4edb866fb2c07959d533: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:24.279927 12560 tablet_bootstrap.cc:492] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02: No bootstrap required, opened a new log
I20260924 03:21:24.280385 12560 ts_tablet_manager.cc:1397] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.009s
I20260924 03:21:24.280529 12561 log.cc:824] T 61cd6359c42a4603bd45df53d3acd75a P 8188e2acd6ea4edb866fb2c07959d533: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:24.281420 12563 tablet_bootstrap.cc:654] T 61cd6359c42a4603bd45df53d3acd75a P 61d88c04b6344d54a3aadac52a4b4ca1: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:24.282615 12563 log.cc:824] T 61cd6359c42a4603bd45df53d3acd75a P 61d88c04b6344d54a3aadac52a4b4ca1: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:24.285744 12563 tablet_bootstrap.cc:492] T 61cd6359c42a4603bd45df53d3acd75a P 61d88c04b6344d54a3aadac52a4b4ca1: No bootstrap required, opened a new log
I20260924 03:21:24.285936 12561 tablet_bootstrap.cc:492] T 61cd6359c42a4603bd45df53d3acd75a P 8188e2acd6ea4edb866fb2c07959d533: No bootstrap required, opened a new log
I20260924 03:21:24.286082 12563 ts_tablet_manager.cc:1397] T 61cd6359c42a4603bd45df53d3acd75a P 61d88c04b6344d54a3aadac52a4b4ca1: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 03:21:24.286254 12561 ts_tablet_manager.cc:1397] T 61cd6359c42a4603bd45df53d3acd75a P 8188e2acd6ea4edb866fb2c07959d533: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.005s
I20260924 03:21:24.292356 12560 raft_consensus.cc:348] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } }
I20260924 03:21:24.294552 12560 raft_consensus.cc:374] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:24.294700 12560 raft_consensus.cc:729] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b7249d472b430ab4dfc3e312d4ea02, State: Initialized, Role: FOLLOWER
I20260924 03:21:24.295527 12560 consensus_queue.cc:260] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } }
I20260924 03:21:24.296007 12563 raft_consensus.cc:348] T 61cd6359c42a4603bd45df53d3acd75a P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } }
I20260924 03:21:24.296543 12563 raft_consensus.cc:374] T 61cd6359c42a4603bd45df53d3acd75a P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:24.296664 12563 raft_consensus.cc:729] T 61cd6359c42a4603bd45df53d3acd75a P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d88c04b6344d54a3aadac52a4b4ca1, State: Initialized, Role: FOLLOWER
I20260924 03:21:24.297241 12541 heartbeater.cc:503] Master 127.10.12.190:39047 was elected leader, sending a full tablet report...
I20260924 03:21:24.297619 12563 consensus_queue.cc:260] T 61cd6359c42a4603bd45df53d3acd75a P 61d88c04b6344d54a3aadac52a4b4ca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } }
I20260924 03:21:24.298743 12402 heartbeater.cc:503] Master 127.10.12.190:39047 was elected leader, sending a full tablet report...
I20260924 03:21:24.299783 12563 ts_tablet_manager.cc:1428] T 61cd6359c42a4603bd45df53d3acd75a P 61d88c04b6344d54a3aadac52a4b4ca1: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260924 03:21:24.299571 12561 raft_consensus.cc:348] T 61cd6359c42a4603bd45df53d3acd75a P 8188e2acd6ea4edb866fb2c07959d533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } }
I20260924 03:21:24.300082 12561 raft_consensus.cc:374] T 61cd6359c42a4603bd45df53d3acd75a P 8188e2acd6ea4edb866fb2c07959d533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:24.300179 12561 raft_consensus.cc:729] T 61cd6359c42a4603bd45df53d3acd75a P 8188e2acd6ea4edb866fb2c07959d533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8188e2acd6ea4edb866fb2c07959d533, State: Initialized, Role: FOLLOWER
I20260924 03:21:24.301182 12561 consensus_queue.cc:260] T 61cd6359c42a4603bd45df53d3acd75a P 8188e2acd6ea4edb866fb2c07959d533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } }
I20260924 03:21:24.304196 12560 ts_tablet_manager.cc:1428] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02: Time spent starting tablet: real 0.024s user 0.012s sys 0.005s
I20260924 03:21:24.304174 12561 ts_tablet_manager.cc:1428] T 61cd6359c42a4603bd45df53d3acd75a P 8188e2acd6ea4edb866fb2c07959d533: Time spent starting tablet: real 0.018s user 0.020s sys 0.000s
W20260924 03:21:24.315903 12403 tablet.cc:2369] T 61cd6359c42a4603bd45df53d3acd75a P 61d88c04b6344d54a3aadac52a4b4ca1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:21:24.373963 12566 raft_consensus.cc:482] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:21:24.374333 12566 raft_consensus.cc:504] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } }
I20260924 03:21:24.376483 12566 leader_election.cc:288] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61d88c04b6344d54a3aadac52a4b4ca1 (127.10.12.130:42369), 8188e2acd6ea4edb866fb2c07959d533 (127.10.12.129:38037)
W20260924 03:21:24.380744 12542 tablet.cc:2369] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:21:24.383586 12430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:24.376962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:38037 (local address 127.10.12.131:48995)
0924 03:21:24.377317 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:24.377337 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:24.377473 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:24.378084 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:24.378091 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:24.378122 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:24.378349 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:24.378362 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.379635 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.379639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:24.381269 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:24.381285 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.382771 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.382776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:24.382985 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:24.383056 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 03:21:24.383150 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 03:21:24.383292 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":2}
I20260924 03:21:24.384205 12559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:24.377356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:48995 (local address 127.10.12.129:38037)
0924 03:21:24.377575 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:24.377581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:24.377600 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:21:24.377663 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:24.377666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:24.377789 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:24.377880 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:24.378516 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.379498 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.381481 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.382423 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.382590 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:24.382669 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:24.383786 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0924 03:21:24.383936 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 03:21:24.385819 12555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:24.376970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:55919 (local address 127.10.12.130:42369)
0924 03:21:24.377204 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:24.377211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:24.378405 (+ 1194us) server_negotiation.cc:407] Connection header received
0924 03:21:24.378582 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:24.378588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:24.378664 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:24.378746 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:24.379394 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.380470 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.383249 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.384550 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.384722 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:24.384796 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:24.385410 (+ 614us) server_negotiation.cc:299] Negotiation successful
0924 03:21:24.385561 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 03:21:24.385458 12213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cd6359c42a4603bd45df53d3acd75a" candidate_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8188e2acd6ea4edb866fb2c07959d533" is_pre_election: true
I20260924 03:21:24.386422 12569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:24.376962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:42369 (local address 127.10.12.131:55919)
0924 03:21:24.378296 (+ 1334us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:24.378320 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:24.378439 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:24.378792 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:24.378797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:24.378912 (+ 115us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:24.379219 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:24.379232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.380649 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.380658 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:24.383044 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:24.383061 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.384775 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.384781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:24.384972 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:24.385007 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 03:21:24.385395 (+ 388us) client_negotiation.cc:240] Negotiation successful
0924 03:21:24.385634 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1183,"mutex_wait_us":396,"thread_start_us":478,"threads_started":1}
I20260924 03:21:24.386461 12213 raft_consensus.cc:2454] T 61cd6359c42a4603bd45df53d3acd75a P 8188e2acd6ea4edb866fb2c07959d533 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0b7249d472b430ab4dfc3e312d4ea02 in term 0.
I20260924 03:21:24.387396 12423 leader_election.cc:302] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8188e2acd6ea4edb866fb2c07959d533, d0b7249d472b430ab4dfc3e312d4ea02; no voters:
I20260924 03:21:24.387579 12356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cd6359c42a4603bd45df53d3acd75a" candidate_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" is_pre_election: true
I20260924 03:21:24.387820 12566 raft_consensus.cc:2781] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:21:24.387995 12566 raft_consensus.cc:482] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:21:24.388103 12566 raft_consensus.cc:3037] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:24.388149 12356 raft_consensus.cc:2454] T 61cd6359c42a4603bd45df53d3acd75a P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0b7249d472b430ab4dfc3e312d4ea02 in term 0.
I20260924 03:21:24.390602 12566 raft_consensus.cc:504] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } }
I20260924 03:21:24.391295 12566 leader_election.cc:288] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02 [CANDIDATE]: Term 1 election: Requested vote from peers 61d88c04b6344d54a3aadac52a4b4ca1 (127.10.12.130:42369), 8188e2acd6ea4edb866fb2c07959d533 (127.10.12.129:38037)
I20260924 03:21:24.391997 12356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cd6359c42a4603bd45df53d3acd75a" candidate_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d88c04b6344d54a3aadac52a4b4ca1"
I20260924 03:21:24.392158 12213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61cd6359c42a4603bd45df53d3acd75a" candidate_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8188e2acd6ea4edb866fb2c07959d533"
I20260924 03:21:24.392221 12356 raft_consensus.cc:3037] T 61cd6359c42a4603bd45df53d3acd75a P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:24.392391 12213 raft_consensus.cc:3037] T 61cd6359c42a4603bd45df53d3acd75a P 8188e2acd6ea4edb866fb2c07959d533 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:24.394629 12356 raft_consensus.cc:2454] T 61cd6359c42a4603bd45df53d3acd75a P 61d88c04b6344d54a3aadac52a4b4ca1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0b7249d472b430ab4dfc3e312d4ea02 in term 1.
I20260924 03:21:24.394798 12213 raft_consensus.cc:2454] T 61cd6359c42a4603bd45df53d3acd75a P 8188e2acd6ea4edb866fb2c07959d533 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0b7249d472b430ab4dfc3e312d4ea02 in term 1.
I20260924 03:21:24.395138 12424 leader_election.cc:302] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61d88c04b6344d54a3aadac52a4b4ca1, d0b7249d472b430ab4dfc3e312d4ea02; no voters:
I20260924 03:21:24.395499 12566 raft_consensus.cc:2781] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:24.395939 12566 raft_consensus.cc:686] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02 [term 1 LEADER]: Becoming Leader. State: Replica: d0b7249d472b430ab4dfc3e312d4ea02, State: Running, Role: LEADER
I20260924 03:21:24.396551 12566 consensus_queue.cc:237] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } }
I20260924 03:21:24.402340 12055 catalog_manager.cc:5495] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0b7249d472b430ab4dfc3e312d4ea02 (127.10.12.131). New cstate: current_term: 1 leader_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } health_report { overall_health: UNKNOWN } } }
Sep 24 03:21:24 dist-test-slave-00qs krb5kdc[12011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220082, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:24.465396 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:24.450831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43494 (local address 127.10.12.190:39047)
0924 03:21:24.451049 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:24.451057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:24.451562 (+ 505us) server_negotiation.cc:407] Connection header received
0924 03:21:24.451728 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:24.451732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:24.451799 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:24.451902 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:24.453335 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.454057 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.455547 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.456068 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.456187 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:24.459780 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:24.459785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:24.459789 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:24.459821 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:24.462419 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:24.463265 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:24.463270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:24.463273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:24.463417 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:24.463774 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:24.463779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:24.463781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:24.464163 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:24.464312 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:24.464736 (+ 424us) server_negotiation.cc:299] Negotiation successful
0924 03:21:24.465081 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":93}
I20260924 03:21:24.465428 12577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:24.450827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39047 (local address 127.0.0.1:43494)
0924 03:21:24.451445 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:24.451465 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:24.451619 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:24.451943 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:24.451947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:24.452870 (+ 923us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:24.453152 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:24.453163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.454200 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.454204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:24.455160 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:24.455289 (+ 129us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.456262 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.456268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:24.456417 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:24.457114 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:24.457137 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:24.459565 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:24.462605 (+ 3040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:24.462623 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:24.462630 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:24.463139 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:24.463527 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:24.463532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:24.463536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:24.463671 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:24.464294 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:24.464303 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:24.464474 (+ 171us) client_negotiation.cc:769] Sending connection context
0924 03:21:24.464623 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 03:21:24.464937 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":348,"threads_started":1}
I20260924 03:21:24.472312 12055 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:43494:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 03:21:24.473536 12055 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:21:24.482395 12193 tablet_service.cc:1467] Processing CreateTablet for tablet c8e1c0d6b7d24fd9ba8dc4c1bfca7560 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=42b1688968d644568b521907fdbb8de2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:21:24.482805 12332 tablet_service.cc:1467] Processing CreateTablet for tablet c8e1c0d6b7d24fd9ba8dc4c1bfca7560 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=42b1688968d644568b521907fdbb8de2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:21:24.482818 12471 tablet_service.cc:1467] Processing CreateTablet for tablet c8e1c0d6b7d24fd9ba8dc4c1bfca7560 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=42b1688968d644568b521907fdbb8de2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:21:24.483568 12332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e1c0d6b7d24fd9ba8dc4c1bfca7560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:24.483582 12193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e1c0d6b7d24fd9ba8dc4c1bfca7560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:24.483582 12471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e1c0d6b7d24fd9ba8dc4c1bfca7560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:24.489019 12563 tablet_bootstrap.cc:492] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 61d88c04b6344d54a3aadac52a4b4ca1: Bootstrap starting.
I20260924 03:21:24.490777 12563 tablet_bootstrap.cc:654] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 61d88c04b6344d54a3aadac52a4b4ca1: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:24.491178 12561 tablet_bootstrap.cc:492] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 8188e2acd6ea4edb866fb2c07959d533: Bootstrap starting.
I20260924 03:21:24.492274 12560 tablet_bootstrap.cc:492] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02: Bootstrap starting.
I20260924 03:21:24.492790 12561 tablet_bootstrap.cc:654] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 8188e2acd6ea4edb866fb2c07959d533: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:24.493439 12563 tablet_bootstrap.cc:492] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 61d88c04b6344d54a3aadac52a4b4ca1: No bootstrap required, opened a new log
I20260924 03:21:24.493641 12563 ts_tablet_manager.cc:1397] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 61d88c04b6344d54a3aadac52a4b4ca1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 03:21:24.493928 12560 tablet_bootstrap.cc:654] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:24.494488 12563 raft_consensus.cc:348] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:24.494781 12563 raft_consensus.cc:374] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:24.494892 12563 raft_consensus.cc:729] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d88c04b6344d54a3aadac52a4b4ca1, State: Initialized, Role: FOLLOWER
I20260924 03:21:24.495286 12563 consensus_queue.cc:260] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 61d88c04b6344d54a3aadac52a4b4ca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:24.495750 12567 tablet_replica.cc:439] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 61d88c04b6344d54a3aadac52a4b4ca1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } } }
I20260924 03:21:24.496160 12567 tablet_replica.cc:442] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 61d88c04b6344d54a3aadac52a4b4ca1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:21:24.497067 12560 tablet_bootstrap.cc:492] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02: No bootstrap required, opened a new log
I20260924 03:21:24.497313 12560 ts_tablet_manager.cc:1397] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260924 03:21:24.497363 12563 ts_tablet_manager.cc:1428] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 61d88c04b6344d54a3aadac52a4b4ca1: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 03:21:24.498247 12560 raft_consensus.cc:348] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:24.498441 12560 raft_consensus.cc:374] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:24.498526 12560 raft_consensus.cc:729] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b7249d472b430ab4dfc3e312d4ea02, State: Initialized, Role: FOLLOWER
I20260924 03:21:24.498755 12560 consensus_queue.cc:260] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:24.499238 12561 tablet_bootstrap.cc:492] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 8188e2acd6ea4edb866fb2c07959d533: No bootstrap required, opened a new log
I20260924 03:21:24.499380 12560 ts_tablet_manager.cc:1428] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 03:21:24.499408 12561 ts_tablet_manager.cc:1397] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 8188e2acd6ea4edb866fb2c07959d533: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 03:21:24.500232 12561 raft_consensus.cc:348] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 8188e2acd6ea4edb866fb2c07959d533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:24.500499 12561 raft_consensus.cc:374] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 8188e2acd6ea4edb866fb2c07959d533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:24.500589 12561 raft_consensus.cc:729] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 8188e2acd6ea4edb866fb2c07959d533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8188e2acd6ea4edb866fb2c07959d533, State: Initialized, Role: FOLLOWER
I20260924 03:21:24.500922 12561 consensus_queue.cc:260] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 8188e2acd6ea4edb866fb2c07959d533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:24.501622 12561 ts_tablet_manager.cc:1428] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 8188e2acd6ea4edb866fb2c07959d533: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:21:24.501518 12568 tablet_replica.cc:439] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 8188e2acd6ea4edb866fb2c07959d533: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } } }
I20260924 03:21:24.501958 12568 tablet_replica.cc:442] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 8188e2acd6ea4edb866fb2c07959d533: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:21:24.499217 12566 tablet_replica.cc:439] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } } }
I20260924 03:21:24.503528 12566 tablet_replica.cc:442] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 03:21:24.519610 12264 tablet.cc:2369] T 61cd6359c42a4603bd45df53d3acd75a P 8188e2acd6ea4edb866fb2c07959d533: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:21:24.566956 12566 raft_consensus.cc:482] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:21:24.567220 12566 raft_consensus.cc:504] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:24.567900 12566 leader_election.cc:288] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8188e2acd6ea4edb866fb2c07959d533 (127.10.12.129:38037), 61d88c04b6344d54a3aadac52a4b4ca1 (127.10.12.130:42369)
I20260924 03:21:24.568550 12356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e1c0d6b7d24fd9ba8dc4c1bfca7560" candidate_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" is_pre_election: true
I20260924 03:21:24.568572 12213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e1c0d6b7d24fd9ba8dc4c1bfca7560" candidate_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8188e2acd6ea4edb866fb2c07959d533" is_pre_election: true
I20260924 03:21:24.568847 12213 raft_consensus.cc:2454] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 8188e2acd6ea4edb866fb2c07959d533 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0b7249d472b430ab4dfc3e312d4ea02 in term 0.
I20260924 03:21:24.568871 12356 raft_consensus.cc:2454] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0b7249d472b430ab4dfc3e312d4ea02 in term 0.
I20260924 03:21:24.569309 12424 leader_election.cc:302] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61d88c04b6344d54a3aadac52a4b4ca1, d0b7249d472b430ab4dfc3e312d4ea02; no voters:
I20260924 03:21:24.569648 12566 raft_consensus.cc:2781] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:21:24.569751 12566 raft_consensus.cc:482] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:21:24.569850 12566 raft_consensus.cc:3037] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:24.571878 12566 raft_consensus.cc:504] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:24.572587 12566 leader_election.cc:288] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02 [CANDIDATE]: Term 1 election: Requested vote from peers 8188e2acd6ea4edb866fb2c07959d533 (127.10.12.129:38037), 61d88c04b6344d54a3aadac52a4b4ca1 (127.10.12.130:42369)
I20260924 03:21:24.573094 12213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e1c0d6b7d24fd9ba8dc4c1bfca7560" candidate_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8188e2acd6ea4edb866fb2c07959d533"
I20260924 03:21:24.573148 12356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e1c0d6b7d24fd9ba8dc4c1bfca7560" candidate_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d88c04b6344d54a3aadac52a4b4ca1"
I20260924 03:21:24.573361 12213 raft_consensus.cc:3037] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 8188e2acd6ea4edb866fb2c07959d533 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:24.573371 12356 raft_consensus.cc:3037] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:24.576370 12356 raft_consensus.cc:2454] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0b7249d472b430ab4dfc3e312d4ea02 in term 1.
I20260924 03:21:24.576380 12213 raft_consensus.cc:2454] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 8188e2acd6ea4edb866fb2c07959d533 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0b7249d472b430ab4dfc3e312d4ea02 in term 1.
I20260924 03:21:24.576925 12424 leader_election.cc:302] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61d88c04b6344d54a3aadac52a4b4ca1, d0b7249d472b430ab4dfc3e312d4ea02; no voters:
I20260924 03:21:24.577260 12566 raft_consensus.cc:2781] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:24.577472 12566 raft_consensus.cc:686] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02 [term 1 LEADER]: Becoming Leader. State: Replica: d0b7249d472b430ab4dfc3e312d4ea02, State: Running, Role: LEADER
I20260924 03:21:24.577770 12566 consensus_queue.cc:237] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:24.579659 12570 tablet_replica.cc:439] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02: TxnStatusTablet state changed. Reason: New leader d0b7249d472b430ab4dfc3e312d4ea02. Latest consensus state: current_term: 1 leader_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } } }
I20260924 03:21:24.580067 12570 tablet_replica.cc:442] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:21:24.580816 12582 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 03:21:24.581135 12582 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 03:21:24.581295 12054 catalog_manager.cc:5495] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0b7249d472b430ab4dfc3e312d4ea02 (127.10.12.131). New cstate: current_term: 1 leader_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } health_report { overall_health: HEALTHY } } }
I20260924 03:21:24.612026 12577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:24.606083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:35321 (local address 127.0.0.1:44270)
0924 03:21:24.606369 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:24.606391 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:24.606555 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:24.607050 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:24.607055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:24.607068 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:21:24.607241 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:24.607250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.608388 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.608393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:24.609403 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:24.609417 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.610150 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.610157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:24.610345 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:24.610411 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:21:24.611369 (+ 958us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:21:24.611393 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 03:21:24.611485 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 03:21:24.611661 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 03:21:24.612011 12556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:24.606243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44270 (local address 127.10.12.131:35321)
0924 03:21:24.606500 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:24.606506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:24.606526 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:21:24.606769 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:24.606773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:24.606874 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:24.607003 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:24.607377 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.608239 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.609552 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.609991 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.610098 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:24.610642 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:24.611256 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:24.611397 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:24.611564 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 03:21:24.611678 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260924 03:21:24.622124 12213 raft_consensus.cc:1261] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 8188e2acd6ea4edb866fb2c07959d533 [term 1 FOLLOWER]: Refusing update from remote peer d0b7249d472b430ab4dfc3e312d4ea02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:21:24.622124 12356 raft_consensus.cc:1261] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 1 FOLLOWER]: Refusing update from remote peer d0b7249d472b430ab4dfc3e312d4ea02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:21:24.623298 12570 consensus_queue.cc:1035] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:21:24.623708 12566 consensus_queue.cc:1035] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:21:24.638168 12567 tablet_replica.cc:439] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 61d88c04b6344d54a3aadac52a4b4ca1: TxnStatusTablet state changed. Reason: New leader d0b7249d472b430ab4dfc3e312d4ea02. Latest consensus state: current_term: 1 leader_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } } }
I20260924 03:21:24.638512 12567 tablet_replica.cc:442] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 61d88c04b6344d54a3aadac52a4b4ca1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:21:24.642469 12566 tablet_replica.cc:439] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } } }
I20260924 03:21:24.642896 12566 tablet_replica.cc:442] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:21:24.643074 12585 mvcc.cc:205] Tried to move back new op lower bound from 7332741466600353792 to 7332741466433769472. Current Snapshot: MvccSnapshot[applied={T|T < 7332741466600353792}]
I20260924 03:21:24.646497 12568 tablet_replica.cc:439] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 8188e2acd6ea4edb866fb2c07959d533: TxnStatusTablet state changed. Reason: New leader d0b7249d472b430ab4dfc3e312d4ea02. Latest consensus state: current_term: 1 leader_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } } }
I20260924 03:21:24.646816 12568 tablet_replica.cc:442] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 8188e2acd6ea4edb866fb2c07959d533: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:21:24.648360 12566 tablet_replica.cc:439] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } } }
I20260924 03:21:24.648666 12566 tablet_replica.cc:442] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P d0b7249d472b430ab4dfc3e312d4ea02: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:21:24.654220 12568 tablet_replica.cc:439] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 8188e2acd6ea4edb866fb2c07959d533: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } } }
I20260924 03:21:24.654531 12568 tablet_replica.cc:442] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 8188e2acd6ea4edb866fb2c07959d533: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:21:24.656769 12567 tablet_replica.cc:439] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 61d88c04b6344d54a3aadac52a4b4ca1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } } }
I20260924 03:21:24.657034 12567 tablet_replica.cc:442] T c8e1c0d6b7d24fd9ba8dc4c1bfca7560 P 61d88c04b6344d54a3aadac52a4b4ca1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:21:24.668423 12556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:24.661325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44286 (local address 127.10.12.131:35321)
0924 03:21:24.661480 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:24.661484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:24.661504 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:21:24.661861 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:24.661867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:24.662161 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:24.662274 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:24.662702 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.663475 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.665603 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.666392 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.666706 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:24.667457 (+ 751us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:24.667635 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:24.667739 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:24.667930 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 03:21:24.668050 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 03:21:24.686823 12556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:24.681037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:52059 (local address 127.10.12.131:35321)
0924 03:21:24.681210 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:24.681216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:24.681692 (+ 476us) server_negotiation.cc:407] Connection header received
0924 03:21:24.681900 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:24.681904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:24.681965 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:24.682035 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:24.682553 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.683282 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.685023 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.685492 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.685611 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:24.686072 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:24.686227 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:24.686300 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:24.686431 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 03:21:24.686508 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 03:21:24.686924 12596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:24.680965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:35321 (local address 127.10.12.131:52059)
0924 03:21:24.681586 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:24.681601 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:24.681760 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:24.682058 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:24.682063 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:24.682073 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:21:24.682404 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:24.682414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.683425 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.683429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:24.684848 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:24.684863 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:24.685694 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:24.685699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:24.685808 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:24.685872 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:21:24.686323 (+ 451us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:21:24.686333 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 03:21:24.686403 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 03:21:24.686557 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":308,"threads_started":1}
I20260924 03:21:24.699465 12356 raft_consensus.cc:1261] T 61cd6359c42a4603bd45df53d3acd75a P 61d88c04b6344d54a3aadac52a4b4ca1 [term 1 FOLLOWER]: Refusing update from remote peer d0b7249d472b430ab4dfc3e312d4ea02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:21:24.699769 12213 raft_consensus.cc:1261] T 61cd6359c42a4603bd45df53d3acd75a P 8188e2acd6ea4edb866fb2c07959d533 [term 1 FOLLOWER]: Refusing update from remote peer d0b7249d472b430ab4dfc3e312d4ea02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:21:24.700707 12597 consensus_queue.cc:1035] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:21:24.701501 12570 consensus_queue.cc:1035] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:21:24.717537 12587 mvcc.cc:205] Tried to move back new op lower bound from 7332741466923503616 to 7332741465693294592. Current Snapshot: MvccSnapshot[applied={T|T < 7332741466923503616}]
I20260924 03:21:24.778074 12054 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:21:24.778344 12054 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:21:24.781919 12054 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366: Sending DeleteTablet for 3 replicas of tablet 61cd6359c42a4603bd45df53d3acd75a
I20260924 03:21:24.783221 12193 tablet_service.cc:1514] Processing DeleteTablet for tablet 61cd6359c42a4603bd45df53d3acd75a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:21:24 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:50114
I20260924 03:21:24.783221 12332 tablet_service.cc:1514] Processing DeleteTablet for tablet 61cd6359c42a4603bd45df53d3acd75a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:21:24 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:40830
I20260924 03:21:24.784155 12471 tablet_service.cc:1514] Processing DeleteTablet for tablet 61cd6359c42a4603bd45df53d3acd75a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:21:24 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:44256
I20260924 03:21:24.785024 12054 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:21:24.785930 12054 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:21:24.788192 12603 tablet_replica.cc:330] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02: stopping tablet replica
I20260924 03:21:24.788808 12603 raft_consensus.cc:2229] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:21:24.789608 12603 raft_consensus.cc:2258] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:21:24.790835 12601 tablet_replica.cc:330] T 61cd6359c42a4603bd45df53d3acd75a P 61d88c04b6344d54a3aadac52a4b4ca1: stopping tablet replica
I20260924 03:21:24.791688 12601 raft_consensus.cc:2229] T 61cd6359c42a4603bd45df53d3acd75a P 61d88c04b6344d54a3aadac52a4b4ca1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:21:24.792330 12601 raft_consensus.cc:2258] T 61cd6359c42a4603bd45df53d3acd75a P 61d88c04b6344d54a3aadac52a4b4ca1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:21:24.793565 12602 tablet_replica.cc:330] T 61cd6359c42a4603bd45df53d3acd75a P 8188e2acd6ea4edb866fb2c07959d533: stopping tablet replica
I20260924 03:21:24.796111 12601 ts_tablet_manager.cc:1905] T 61cd6359c42a4603bd45df53d3acd75a P 61d88c04b6344d54a3aadac52a4b4ca1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:21:24.796209 12603 ts_tablet_manager.cc:1905] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:21:24.796111 12471 tablet_service.cc:1467] Processing CreateTablet for tablet c6ff2be4d51d466d86fafb2028d8cc36 (DEFAULT_TABLE table=test-table [id=85fe165ad990461688063c5e78eabdb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:24.796636 12471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ff2be4d51d466d86fafb2028d8cc36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:24.800506 12602 raft_consensus.cc:2229] T 61cd6359c42a4603bd45df53d3acd75a P 8188e2acd6ea4edb866fb2c07959d533 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:21:24.801110 12602 raft_consensus.cc:2258] T 61cd6359c42a4603bd45df53d3acd75a P 8188e2acd6ea4edb866fb2c07959d533 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:21:24.801534 12193 tablet_service.cc:1467] Processing CreateTablet for tablet c6ff2be4d51d466d86fafb2028d8cc36 (DEFAULT_TABLE table=test-table [id=85fe165ad990461688063c5e78eabdb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:24.802028 12193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ff2be4d51d466d86fafb2028d8cc36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:24.802784 12332 tablet_service.cc:1467] Processing CreateTablet for tablet c6ff2be4d51d466d86fafb2028d8cc36 (DEFAULT_TABLE table=test-table [id=85fe165ad990461688063c5e78eabdb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:24.803251 12332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ff2be4d51d466d86fafb2028d8cc36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:24.804095 12602 ts_tablet_manager.cc:1905] T 61cd6359c42a4603bd45df53d3acd75a P 8188e2acd6ea4edb866fb2c07959d533: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:21:24.810395 12563 tablet_bootstrap.cc:492] T c6ff2be4d51d466d86fafb2028d8cc36 P 61d88c04b6344d54a3aadac52a4b4ca1: Bootstrap starting.
I20260924 03:21:24.812124 12563 tablet_bootstrap.cc:654] T c6ff2be4d51d466d86fafb2028d8cc36 P 61d88c04b6344d54a3aadac52a4b4ca1: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:24.812918 12601 ts_tablet_manager.cc:1918] T 61cd6359c42a4603bd45df53d3acd75a P 61d88c04b6344d54a3aadac52a4b4ca1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:21:24.813077 12601 log.cc:1196] T 61cd6359c42a4603bd45df53d3acd75a P 61d88c04b6344d54a3aadac52a4b4ca1: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/61cd6359c42a4603bd45df53d3acd75a
I20260924 03:21:24.813635 12601 ts_tablet_manager.cc:1939] T 61cd6359c42a4603bd45df53d3acd75a P 61d88c04b6344d54a3aadac52a4b4ca1: Deleting consensus metadata
I20260924 03:21:24.814597 12603 ts_tablet_manager.cc:1918] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:21:24.814668 12563 tablet_bootstrap.cc:492] T c6ff2be4d51d466d86fafb2028d8cc36 P 61d88c04b6344d54a3aadac52a4b4ca1: No bootstrap required, opened a new log
I20260924 03:21:24.814808 12603 log.cc:1196] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/61cd6359c42a4603bd45df53d3acd75a
I20260924 03:21:24.814828 12563 ts_tablet_manager.cc:1397] T c6ff2be4d51d466d86fafb2028d8cc36 P 61d88c04b6344d54a3aadac52a4b4ca1: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260924 03:21:24.815340 12603 ts_tablet_manager.cc:1939] T 61cd6359c42a4603bd45df53d3acd75a P d0b7249d472b430ab4dfc3e312d4ea02: Deleting consensus metadata
I20260924 03:21:24.815629 12563 raft_consensus.cc:348] T c6ff2be4d51d466d86fafb2028d8cc36 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:24.815874 12563 raft_consensus.cc:374] T c6ff2be4d51d466d86fafb2028d8cc36 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:24.815969 12563 raft_consensus.cc:729] T c6ff2be4d51d466d86fafb2028d8cc36 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d88c04b6344d54a3aadac52a4b4ca1, State: Initialized, Role: FOLLOWER
I20260924 03:21:24.816273 12563 consensus_queue.cc:260] T c6ff2be4d51d466d86fafb2028d8cc36 P 61d88c04b6344d54a3aadac52a4b4ca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:24.816798 12042 catalog_manager.cc:4841] TS 61d88c04b6344d54a3aadac52a4b4ca1 (127.10.12.130:42369): tablet 61cd6359c42a4603bd45df53d3acd75a (table test-table [id=dc7c4dbc46e04a98a647307af2f38c15]) successfully deleted
I20260924 03:21:24.818473 12563 ts_tablet_manager.cc:1428] T c6ff2be4d51d466d86fafb2028d8cc36 P 61d88c04b6344d54a3aadac52a4b4ca1: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 03:21:24.819365 12602 ts_tablet_manager.cc:1918] T 61cd6359c42a4603bd45df53d3acd75a P 8188e2acd6ea4edb866fb2c07959d533: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:21:24.819589 12602 log.cc:1196] T 61cd6359c42a4603bd45df53d3acd75a P 8188e2acd6ea4edb866fb2c07959d533: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-0/wal/wals/61cd6359c42a4603bd45df53d3acd75a
I20260924 03:21:24.820139 12602 ts_tablet_manager.cc:1939] T 61cd6359c42a4603bd45df53d3acd75a P 8188e2acd6ea4edb866fb2c07959d533: Deleting consensus metadata
I20260924 03:21:24.821962 12041 catalog_manager.cc:4841] TS 8188e2acd6ea4edb866fb2c07959d533 (127.10.12.129:38037): tablet 61cd6359c42a4603bd45df53d3acd75a (table test-table [id=dc7c4dbc46e04a98a647307af2f38c15]) successfully deleted
I20260924 03:21:24.822768 12043 catalog_manager.cc:4841] TS d0b7249d472b430ab4dfc3e312d4ea02 (127.10.12.131:35321): tablet 61cd6359c42a4603bd45df53d3acd75a (table test-table [id=dc7c4dbc46e04a98a647307af2f38c15]) successfully deleted
I20260924 03:21:24.823127 12561 tablet_bootstrap.cc:492] T c6ff2be4d51d466d86fafb2028d8cc36 P 8188e2acd6ea4edb866fb2c07959d533: Bootstrap starting.
I20260924 03:21:24.824535 12560 tablet_bootstrap.cc:492] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02: Bootstrap starting.
I20260924 03:21:24.824851 12561 tablet_bootstrap.cc:654] T c6ff2be4d51d466d86fafb2028d8cc36 P 8188e2acd6ea4edb866fb2c07959d533: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:24.826218 12560 tablet_bootstrap.cc:654] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:24.827205 12561 tablet_bootstrap.cc:492] T c6ff2be4d51d466d86fafb2028d8cc36 P 8188e2acd6ea4edb866fb2c07959d533: No bootstrap required, opened a new log
I20260924 03:21:24.827461 12561 ts_tablet_manager.cc:1397] T c6ff2be4d51d466d86fafb2028d8cc36 P 8188e2acd6ea4edb866fb2c07959d533: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260924 03:21:24.828285 12561 raft_consensus.cc:348] T c6ff2be4d51d466d86fafb2028d8cc36 P 8188e2acd6ea4edb866fb2c07959d533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:24.828548 12561 raft_consensus.cc:374] T c6ff2be4d51d466d86fafb2028d8cc36 P 8188e2acd6ea4edb866fb2c07959d533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:24.828629 12561 raft_consensus.cc:729] T c6ff2be4d51d466d86fafb2028d8cc36 P 8188e2acd6ea4edb866fb2c07959d533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8188e2acd6ea4edb866fb2c07959d533, State: Initialized, Role: FOLLOWER
I20260924 03:21:24.828926 12561 consensus_queue.cc:260] T c6ff2be4d51d466d86fafb2028d8cc36 P 8188e2acd6ea4edb866fb2c07959d533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:24.830068 12561 ts_tablet_manager.cc:1428] T c6ff2be4d51d466d86fafb2028d8cc36 P 8188e2acd6ea4edb866fb2c07959d533: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:21:24.831934 12560 tablet_bootstrap.cc:492] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02: No bootstrap required, opened a new log
I20260924 03:21:24.832165 12560 ts_tablet_manager.cc:1397] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260924 03:21:24.832981 12560 raft_consensus.cc:348] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:24.833245 12560 raft_consensus.cc:374] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:24.833312 12560 raft_consensus.cc:729] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b7249d472b430ab4dfc3e312d4ea02, State: Initialized, Role: FOLLOWER
I20260924 03:21:24.833570 12560 consensus_queue.cc:260] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:24.834163 12560 ts_tablet_manager.cc:1428] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 03:21:24.845134 12570 raft_consensus.cc:482] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:21:24.845356 12570 raft_consensus.cc:504] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:24.846015 12570 leader_election.cc:288] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8188e2acd6ea4edb866fb2c07959d533 (127.10.12.129:38037), 61d88c04b6344d54a3aadac52a4b4ca1 (127.10.12.130:42369)
I20260924 03:21:24.846613 12355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ff2be4d51d466d86fafb2028d8cc36" candidate_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" is_pre_election: true
I20260924 03:21:24.846616 12212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ff2be4d51d466d86fafb2028d8cc36" candidate_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8188e2acd6ea4edb866fb2c07959d533" is_pre_election: true
I20260924 03:21:24.846940 12355 raft_consensus.cc:2454] T c6ff2be4d51d466d86fafb2028d8cc36 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0b7249d472b430ab4dfc3e312d4ea02 in term 0.
I20260924 03:21:24.846974 12212 raft_consensus.cc:2454] T c6ff2be4d51d466d86fafb2028d8cc36 P 8188e2acd6ea4edb866fb2c07959d533 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0b7249d472b430ab4dfc3e312d4ea02 in term 0.
I20260924 03:21:24.847450 12424 leader_election.cc:302] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61d88c04b6344d54a3aadac52a4b4ca1, d0b7249d472b430ab4dfc3e312d4ea02; no voters:
I20260924 03:21:24.847869 12570 raft_consensus.cc:2781] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:21:24.848011 12570 raft_consensus.cc:482] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:21:24.848101 12570 raft_consensus.cc:3037] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:24.851089 12570 raft_consensus.cc:504] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:24.852613 12212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ff2be4d51d466d86fafb2028d8cc36" candidate_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8188e2acd6ea4edb866fb2c07959d533"
I20260924 03:21:24.852866 12212 raft_consensus.cc:3037] T c6ff2be4d51d466d86fafb2028d8cc36 P 8188e2acd6ea4edb866fb2c07959d533 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:24.853621 12570 leader_election.cc:288] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02 [CANDIDATE]: Term 1 election: Requested vote from peers 8188e2acd6ea4edb866fb2c07959d533 (127.10.12.129:38037), 61d88c04b6344d54a3aadac52a4b4ca1 (127.10.12.130:42369)
I20260924 03:21:24.854216 12355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ff2be4d51d466d86fafb2028d8cc36" candidate_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d88c04b6344d54a3aadac52a4b4ca1"
I20260924 03:21:24.854431 12355 raft_consensus.cc:3037] T c6ff2be4d51d466d86fafb2028d8cc36 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:24.856149 12212 raft_consensus.cc:2454] T c6ff2be4d51d466d86fafb2028d8cc36 P 8188e2acd6ea4edb866fb2c07959d533 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0b7249d472b430ab4dfc3e312d4ea02 in term 1.
I20260924 03:21:24.856683 12355 raft_consensus.cc:2454] T c6ff2be4d51d466d86fafb2028d8cc36 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0b7249d472b430ab4dfc3e312d4ea02 in term 1.
I20260924 03:21:24.856698 12423 leader_election.cc:302] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8188e2acd6ea4edb866fb2c07959d533, d0b7249d472b430ab4dfc3e312d4ea02; no voters:
I20260924 03:21:24.857081 12570 raft_consensus.cc:2781] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:24.857234 12570 raft_consensus.cc:686] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02 [term 1 LEADER]: Becoming Leader. State: Replica: d0b7249d472b430ab4dfc3e312d4ea02, State: Running, Role: LEADER
I20260924 03:21:24.857525 12570 consensus_queue.cc:237] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:24.860374 12054 catalog_manager.cc:5495] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0b7249d472b430ab4dfc3e312d4ea02 (127.10.12.131). New cstate: current_term: 1 leader_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } health_report { overall_health: HEALTHY } } }
I20260924 03:21:24.892006 12212 raft_consensus.cc:1261] T c6ff2be4d51d466d86fafb2028d8cc36 P 8188e2acd6ea4edb866fb2c07959d533 [term 1 FOLLOWER]: Refusing update from remote peer d0b7249d472b430ab4dfc3e312d4ea02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:21:24.892127 12355 raft_consensus.cc:1261] T c6ff2be4d51d466d86fafb2028d8cc36 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 1 FOLLOWER]: Refusing update from remote peer d0b7249d472b430ab4dfc3e312d4ea02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:21:24.892690 12597 consensus_queue.cc:1035] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:21:24.892971 12570 consensus_queue.cc:1035] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 03:21:24.921766 12585 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 03:21:26.522282 12054 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:21:26.522594 12054 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:21:26.527832 12054 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366: Sending DeleteTablet for 3 replicas of tablet c6ff2be4d51d466d86fafb2028d8cc36
I20260924 03:21:26.529006 12332 tablet_service.cc:1514] Processing DeleteTablet for tablet c6ff2be4d51d466d86fafb2028d8cc36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:21:26 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:40830
I20260924 03:21:26.529035 12193 tablet_service.cc:1514] Processing DeleteTablet for tablet c6ff2be4d51d466d86fafb2028d8cc36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:21:26 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:50114
I20260924 03:21:26.529049 12471 tablet_service.cc:1514] Processing DeleteTablet for tablet c6ff2be4d51d466d86fafb2028d8cc36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:21:26 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:44256
I20260924 03:21:26.529838 12631 tablet_replica.cc:330] T c6ff2be4d51d466d86fafb2028d8cc36 P 61d88c04b6344d54a3aadac52a4b4ca1: stopping tablet replica
I20260924 03:21:26.530146 12631 raft_consensus.cc:2229] T c6ff2be4d51d466d86fafb2028d8cc36 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:21:26.530529 12631 raft_consensus.cc:2258] T c6ff2be4d51d466d86fafb2028d8cc36 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:21:26.531903 12631 ts_tablet_manager.cc:1905] T c6ff2be4d51d466d86fafb2028d8cc36 P 61d88c04b6344d54a3aadac52a4b4ca1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:21:26.541113 12631 ts_tablet_manager.cc:1918] T c6ff2be4d51d466d86fafb2028d8cc36 P 61d88c04b6344d54a3aadac52a4b4ca1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:21:26.541531 12631 log.cc:1196] T c6ff2be4d51d466d86fafb2028d8cc36 P 61d88c04b6344d54a3aadac52a4b4ca1: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/c6ff2be4d51d466d86fafb2028d8cc36
I20260924 03:21:26.542186 12631 ts_tablet_manager.cc:1939] T c6ff2be4d51d466d86fafb2028d8cc36 P 61d88c04b6344d54a3aadac52a4b4ca1: Deleting consensus metadata
I20260924 03:21:26.543823 12042 catalog_manager.cc:4841] TS 61d88c04b6344d54a3aadac52a4b4ca1 (127.10.12.130:42369): tablet c6ff2be4d51d466d86fafb2028d8cc36 (table test-table [id=85fe165ad990461688063c5e78eabdb9]) successfully deleted
I20260924 03:21:26.550263 12630 tablet_replica.cc:330] T c6ff2be4d51d466d86fafb2028d8cc36 P 8188e2acd6ea4edb866fb2c07959d533: stopping tablet replica
I20260924 03:21:26.551419 12630 raft_consensus.cc:2229] T c6ff2be4d51d466d86fafb2028d8cc36 P 8188e2acd6ea4edb866fb2c07959d533 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:21:26.551483 12629 tablet_replica.cc:330] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02: stopping tablet replica
I20260924 03:21:26.551735 12629 raft_consensus.cc:2229] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:21:26.552022 12630 raft_consensus.cc:2258] T c6ff2be4d51d466d86fafb2028d8cc36 P 8188e2acd6ea4edb866fb2c07959d533 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:21:26.552241 12629 raft_consensus.cc:2258] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:21:26.554317 12630 ts_tablet_manager.cc:1905] T c6ff2be4d51d466d86fafb2028d8cc36 P 8188e2acd6ea4edb866fb2c07959d533: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:21:26.554574 12629 ts_tablet_manager.cc:1905] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:21:26.561723 12630 ts_tablet_manager.cc:1918] T c6ff2be4d51d466d86fafb2028d8cc36 P 8188e2acd6ea4edb866fb2c07959d533: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:21:26.562021 12630 log.cc:1196] T c6ff2be4d51d466d86fafb2028d8cc36 P 8188e2acd6ea4edb866fb2c07959d533: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-0/wal/wals/c6ff2be4d51d466d86fafb2028d8cc36
I20260924 03:21:26.562088 12629 ts_tablet_manager.cc:1918] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:21:26.562258 12629 log.cc:1196] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/c6ff2be4d51d466d86fafb2028d8cc36
I20260924 03:21:26.562528 12630 ts_tablet_manager.cc:1939] T c6ff2be4d51d466d86fafb2028d8cc36 P 8188e2acd6ea4edb866fb2c07959d533: Deleting consensus metadata
I20260924 03:21:26.562734 12629 ts_tablet_manager.cc:1939] T c6ff2be4d51d466d86fafb2028d8cc36 P d0b7249d472b430ab4dfc3e312d4ea02: Deleting consensus metadata
I20260924 03:21:26.564324 12041 catalog_manager.cc:4841] TS 8188e2acd6ea4edb866fb2c07959d533 (127.10.12.129:38037): tablet c6ff2be4d51d466d86fafb2028d8cc36 (table test-table [id=85fe165ad990461688063c5e78eabdb9]) successfully deleted
I20260924 03:21:26.564324 12043 catalog_manager.cc:4841] TS d0b7249d472b430ab4dfc3e312d4ea02 (127.10.12.131:35321): tablet c6ff2be4d51d466d86fafb2028d8cc36 (table test-table [id=85fe165ad990461688063c5e78eabdb9]) successfully deleted
Sep 24 03:21:26 dist-test-slave-00qs krb5kdc[12011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220086, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 03:21:26 dist-test-slave-00qs krb5kdc[12011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220086, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:26.602139 12639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:26.582446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43500 (local address 127.10.12.190:39047)
0924 03:21:26.583210 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:26.583217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:26.583337 (+ 120us) server_negotiation.cc:407] Connection header received
0924 03:21:26.583580 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:26.583585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:26.583670 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:26.583776 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:26.585638 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:26.586525 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:26.588036 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:26.588513 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:26.588647 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:26.594358 (+ 5711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:26.594469 (+ 111us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:26.594479 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:26.594558 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:26.598354 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:26.599498 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:26.599503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:26.599506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:26.599726 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:26.600268 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:26.600274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:26.600276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:26.600647 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:26.600888 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:26.601346 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 03:21:26.601646 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":402,"threads_started":1}
I20260924 03:21:26.607836 12054 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:21:26.608875 12054 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:21:26.617151 12193 tablet_service.cc:1467] Processing CreateTablet for tablet 05f9a54a02104f4c98caab7fbdf39e74 (DEFAULT_TABLE table=test-table [id=76cd403927a1431c87d29a764f4a8676]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:26.617175 12332 tablet_service.cc:1467] Processing CreateTablet for tablet 05f9a54a02104f4c98caab7fbdf39e74 (DEFAULT_TABLE table=test-table [id=76cd403927a1431c87d29a764f4a8676]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:26.617822 12193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05f9a54a02104f4c98caab7fbdf39e74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:26.617822 12332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05f9a54a02104f4c98caab7fbdf39e74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:26.618000 12471 tablet_service.cc:1467] Processing CreateTablet for tablet 05f9a54a02104f4c98caab7fbdf39e74 (DEFAULT_TABLE table=test-table [id=76cd403927a1431c87d29a764f4a8676]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:26.618489 12471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05f9a54a02104f4c98caab7fbdf39e74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:26.625377 12640 tablet_bootstrap.cc:492] T 05f9a54a02104f4c98caab7fbdf39e74 P d0b7249d472b430ab4dfc3e312d4ea02: Bootstrap starting.
I20260924 03:21:26.626217 12641 tablet_bootstrap.cc:492] T 05f9a54a02104f4c98caab7fbdf39e74 P 8188e2acd6ea4edb866fb2c07959d533: Bootstrap starting.
I20260924 03:21:26.627131 12642 tablet_bootstrap.cc:492] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1: Bootstrap starting.
I20260924 03:21:26.629094 12641 tablet_bootstrap.cc:654] T 05f9a54a02104f4c98caab7fbdf39e74 P 8188e2acd6ea4edb866fb2c07959d533: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:26.629163 12642 tablet_bootstrap.cc:654] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:26.629151 12640 tablet_bootstrap.cc:654] T 05f9a54a02104f4c98caab7fbdf39e74 P d0b7249d472b430ab4dfc3e312d4ea02: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:26.631729 12641 tablet_bootstrap.cc:492] T 05f9a54a02104f4c98caab7fbdf39e74 P 8188e2acd6ea4edb866fb2c07959d533: No bootstrap required, opened a new log
I20260924 03:21:26.631759 12640 tablet_bootstrap.cc:492] T 05f9a54a02104f4c98caab7fbdf39e74 P d0b7249d472b430ab4dfc3e312d4ea02: No bootstrap required, opened a new log
I20260924 03:21:26.631913 12641 ts_tablet_manager.cc:1397] T 05f9a54a02104f4c98caab7fbdf39e74 P 8188e2acd6ea4edb866fb2c07959d533: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 03:21:26.631929 12640 ts_tablet_manager.cc:1397] T 05f9a54a02104f4c98caab7fbdf39e74 P d0b7249d472b430ab4dfc3e312d4ea02: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260924 03:21:26.632537 12642 tablet_bootstrap.cc:492] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1: No bootstrap required, opened a new log
I20260924 03:21:26.632714 12642 ts_tablet_manager.cc:1397] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 03:21:26.632877 12640 raft_consensus.cc:348] T 05f9a54a02104f4c98caab7fbdf39e74 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:26.632881 12641 raft_consensus.cc:348] T 05f9a54a02104f4c98caab7fbdf39e74 P 8188e2acd6ea4edb866fb2c07959d533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:26.633174 12640 raft_consensus.cc:374] T 05f9a54a02104f4c98caab7fbdf39e74 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:26.633193 12641 raft_consensus.cc:374] T 05f9a54a02104f4c98caab7fbdf39e74 P 8188e2acd6ea4edb866fb2c07959d533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:26.633270 12640 raft_consensus.cc:729] T 05f9a54a02104f4c98caab7fbdf39e74 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b7249d472b430ab4dfc3e312d4ea02, State: Initialized, Role: FOLLOWER
I20260924 03:21:26.633288 12641 raft_consensus.cc:729] T 05f9a54a02104f4c98caab7fbdf39e74 P 8188e2acd6ea4edb866fb2c07959d533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8188e2acd6ea4edb866fb2c07959d533, State: Initialized, Role: FOLLOWER
I20260924 03:21:26.633535 12641 consensus_queue.cc:260] T 05f9a54a02104f4c98caab7fbdf39e74 P 8188e2acd6ea4edb866fb2c07959d533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:26.633580 12640 consensus_queue.cc:260] T 05f9a54a02104f4c98caab7fbdf39e74 P d0b7249d472b430ab4dfc3e312d4ea02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:26.633553 12642 raft_consensus.cc:348] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:26.633821 12642 raft_consensus.cc:374] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:26.633914 12642 raft_consensus.cc:729] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d88c04b6344d54a3aadac52a4b4ca1, State: Initialized, Role: FOLLOWER
I20260924 03:21:26.634266 12640 ts_tablet_manager.cc:1428] T 05f9a54a02104f4c98caab7fbdf39e74 P d0b7249d472b430ab4dfc3e312d4ea02: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:21:26.634145 12642 consensus_queue.cc:260] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:26.635185 12642 ts_tablet_manager.cc:1428] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 03:21:26.636492 12641 ts_tablet_manager.cc:1428] T 05f9a54a02104f4c98caab7fbdf39e74 P 8188e2acd6ea4edb866fb2c07959d533: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 03:21:26.692606 12646 raft_consensus.cc:482] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:21:26.692917 12646 raft_consensus.cc:504] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:26.695016 12646 leader_election.cc:288] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8188e2acd6ea4edb866fb2c07959d533 (127.10.12.129:38037), d0b7249d472b430ab4dfc3e312d4ea02 (127.10.12.131:35321)
I20260924 03:21:26.715083 12651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:26.695561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:49951 (local address 127.10.12.129:38037)
0924 03:21:26.707392 (+ 11831us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:26.707399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:26.707882 (+ 483us) server_negotiation.cc:407] Connection header received
0924 03:21:26.708163 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:26.708167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:26.708240 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:26.708338 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:26.709073 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:26.709968 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:26.712576 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:26.713954 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:26.714155 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:26.714227 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:26.714626 (+ 399us) server_negotiation.cc:299] Negotiation successful
0924 03:21:26.714759 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11641,"thread_start_us":12136,"threads_started":1}
I20260924 03:21:26.715204 12648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:26.695487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:38037 (local address 127.10.12.130:49951)
0924 03:21:26.707659 (+ 12172us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:26.707684 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:26.707875 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:26.708390 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:26.708395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:26.708422 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:26.708862 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:26.708877 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:26.710213 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:26.710225 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:26.712361 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:26.712377 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:26.714204 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:26.714212 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:26.714422 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:26.714463 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 03:21:26.714585 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 03:21:26.714775 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12046,"mutex_wait_us":28,"thread_start_us":11404,"threads_started":1}
I20260924 03:21:26.716077 12212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f9a54a02104f4c98caab7fbdf39e74" candidate_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8188e2acd6ea4edb866fb2c07959d533" is_pre_election: true
I20260924 03:21:26.716415 12212 raft_consensus.cc:2454] T 05f9a54a02104f4c98caab7fbdf39e74 P 8188e2acd6ea4edb866fb2c07959d533 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61d88c04b6344d54a3aadac52a4b4ca1 in term 0.
I20260924 03:21:26.716584 12649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:26.695374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:35321 (local address 127.10.12.130:38575)
0924 03:21:26.707322 (+ 11948us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:26.707353 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:26.707588 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:26.708585 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:26.708588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:26.708613 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:26.709076 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:26.709088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:26.711668 (+ 2580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:26.711672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:26.713866 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:26.713885 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:26.715759 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:26.715766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:26.715923 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:26.715980 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 03:21:26.716087 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 03:21:26.716211 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11723,"mutex_wait_us":2,"thread_start_us":11975,"threads_started":1}
I20260924 03:21:26.716645 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:26.695561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:38575 (local address 127.10.12.131:35321)
0924 03:21:26.708217 (+ 12656us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:26.708222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:26.708246 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:21:26.708321 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:26.708326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:26.708405 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:26.708532 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:26.709325 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:26.711491 (+ 2166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:26.714136 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:26.715523 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:26.715738 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:26.715809 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:26.716147 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 03:21:26.716308 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12486,"thread_start_us":12241,"threads_started":1}
I20260924 03:21:26.717082 12284 leader_election.cc:302] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61d88c04b6344d54a3aadac52a4b4ca1, 8188e2acd6ea4edb866fb2c07959d533; no voters:
I20260924 03:21:26.717571 12646 raft_consensus.cc:2781] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:21:26.717716 12646 raft_consensus.cc:482] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:21:26.717830 12646 raft_consensus.cc:3037] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:26.717666 12491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f9a54a02104f4c98caab7fbdf39e74" candidate_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" is_pre_election: true
I20260924 03:21:26.718163 12491 raft_consensus.cc:2454] T 05f9a54a02104f4c98caab7fbdf39e74 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61d88c04b6344d54a3aadac52a4b4ca1 in term 0.
I20260924 03:21:26.720901 12646 raft_consensus.cc:504] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:26.721772 12646 leader_election.cc:288] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1 [CANDIDATE]: Term 1 election: Requested vote from peers 8188e2acd6ea4edb866fb2c07959d533 (127.10.12.129:38037), d0b7249d472b430ab4dfc3e312d4ea02 (127.10.12.131:35321)
I20260924 03:21:26.722434 12212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f9a54a02104f4c98caab7fbdf39e74" candidate_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8188e2acd6ea4edb866fb2c07959d533"
I20260924 03:21:26.722431 12491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f9a54a02104f4c98caab7fbdf39e74" candidate_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0b7249d472b430ab4dfc3e312d4ea02"
I20260924 03:21:26.722736 12491 raft_consensus.cc:3037] T 05f9a54a02104f4c98caab7fbdf39e74 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:26.722734 12212 raft_consensus.cc:3037] T 05f9a54a02104f4c98caab7fbdf39e74 P 8188e2acd6ea4edb866fb2c07959d533 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:26.725914 12212 raft_consensus.cc:2454] T 05f9a54a02104f4c98caab7fbdf39e74 P 8188e2acd6ea4edb866fb2c07959d533 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61d88c04b6344d54a3aadac52a4b4ca1 in term 1.
I20260924 03:21:26.726286 12491 raft_consensus.cc:2454] T 05f9a54a02104f4c98caab7fbdf39e74 P d0b7249d472b430ab4dfc3e312d4ea02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61d88c04b6344d54a3aadac52a4b4ca1 in term 1.
I20260924 03:21:26.726504 12284 leader_election.cc:302] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61d88c04b6344d54a3aadac52a4b4ca1, 8188e2acd6ea4edb866fb2c07959d533; no voters:
I20260924 03:21:26.726886 12646 raft_consensus.cc:2781] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:26.727425 12646 raft_consensus.cc:686] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 1 LEADER]: Becoming Leader. State: Replica: 61d88c04b6344d54a3aadac52a4b4ca1, State: Running, Role: LEADER
I20260924 03:21:26.727871 12646 consensus_queue.cc:237] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:26.732275 12054 catalog_manager.cc:5495] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61d88c04b6344d54a3aadac52a4b4ca1 (127.10.12.130). New cstate: current_term: 1 leader_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } health_report { overall_health: UNKNOWN } } }
I20260924 03:21:26.774884 12656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:26.768037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40834 (local address 127.10.12.130:42369)
0924 03:21:26.768855 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:26.768860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:26.768882 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:21:26.769044 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:26.769048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:26.769149 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:26.769331 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:26.769845 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:26.770837 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:26.772287 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:26.772747 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:26.772848 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:26.773336 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:26.774162 (+ 826us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:26.774290 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:26.774374 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 03:21:26.774466 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":547,"threads_started":1}
I20260924 03:21:26.794579 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:26.788839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:50831 (local address 127.10.12.131:35321)
0924 03:21:26.789063 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:26.789068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:26.789518 (+ 450us) server_negotiation.cc:407] Connection header received
0924 03:21:26.789735 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:26.789740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:26.789825 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:26.789926 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:26.790489 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:26.791325 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:26.792826 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:26.793263 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:26.793373 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:26.793873 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:26.794003 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:26.794088 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:26.794207 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 03:21:26.794292 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 03:21:26.794622 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:26.788667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:35321 (local address 127.10.12.130:50831)
0924 03:21:26.789386 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:26.789404 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:26.789556 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:26.789944 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:26.789948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:26.789958 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:21:26.790327 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:26.790336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:26.791465 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:26.791470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:26.792676 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:26.792690 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:26.793429 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:26.793433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:26.793587 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:26.793665 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:21:26.794102 (+ 437us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:21:26.794118 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 03:21:26.794181 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 03:21:26.794302 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":333,"threads_started":1}
I20260924 03:21:26.805923 12491 raft_consensus.cc:1261] T 05f9a54a02104f4c98caab7fbdf39e74 P d0b7249d472b430ab4dfc3e312d4ea02 [term 1 FOLLOWER]: Refusing update from remote peer 61d88c04b6344d54a3aadac52a4b4ca1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:21:26.806306 12212 raft_consensus.cc:1261] T 05f9a54a02104f4c98caab7fbdf39e74 P 8188e2acd6ea4edb866fb2c07959d533 [term 1 FOLLOWER]: Refusing update from remote peer 61d88c04b6344d54a3aadac52a4b4ca1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:21:26.807013 12646 consensus_queue.cc:1035] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:21:26.807394 12652 consensus_queue.cc:1035] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:21:26.814829 12613 mvcc.cc:205] Tried to move back new op lower bound from 7332741475547021312 to 7332741475242139648. Current Snapshot: MvccSnapshot[applied={T|T < 7332741475547021312}]
I20260924 03:21:26.840735 12656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:26.834898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:44011 (local address 127.10.12.130:42369)
0924 03:21:26.835267 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:26.835274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:26.835570 (+ 296us) server_negotiation.cc:407] Connection header received
0924 03:21:26.835778 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:26.835784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:26.835868 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:26.835943 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:26.836471 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:26.837244 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:26.838795 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:26.839347 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:26.839458 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:26.839878 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:26.840059 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:26.840136 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:26.840289 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 03:21:26.840381 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260924 03:21:26.840802 12664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:26.834730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:42369 (local address 127.10.12.131:44011)
0924 03:21:26.835467 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:26.835484 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:26.835635 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:26.836016 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:26.836020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:26.836034 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:21:26.836334 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:26.836344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:26.837379 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:26.837384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:26.838662 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:26.838674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:26.839517 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:26.839524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:26.839674 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:26.839718 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:21:26.840174 (+ 456us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:21:26.840185 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 03:21:26.840257 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 03:21:26.840415 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":438,"threads_started":1}
I20260924 03:21:26.884115 12054 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:21:26.884528 12054 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:21:26.889161 12054 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366: Sending DeleteTablet for 3 replicas of tablet 05f9a54a02104f4c98caab7fbdf39e74
I20260924 03:21:26.890635 12471 tablet_service.cc:1514] Processing DeleteTablet for tablet 05f9a54a02104f4c98caab7fbdf39e74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:21:26 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:44256
I20260924 03:21:26.890615 12332 tablet_service.cc:1514] Processing DeleteTablet for tablet 05f9a54a02104f4c98caab7fbdf39e74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:21:26 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:40830
I20260924 03:21:26.890612 12193 tablet_service.cc:1514] Processing DeleteTablet for tablet 05f9a54a02104f4c98caab7fbdf39e74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:21:26 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:50114
I20260924 03:21:26.891085 12630 tablet_replica.cc:330] T 05f9a54a02104f4c98caab7fbdf39e74 P 8188e2acd6ea4edb866fb2c07959d533: stopping tablet replica
I20260924 03:21:26.891114 12629 tablet_replica.cc:330] T 05f9a54a02104f4c98caab7fbdf39e74 P d0b7249d472b430ab4dfc3e312d4ea02: stopping tablet replica
I20260924 03:21:26.891333 12629 raft_consensus.cc:2229] T 05f9a54a02104f4c98caab7fbdf39e74 P d0b7249d472b430ab4dfc3e312d4ea02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:21:26.891327 12630 raft_consensus.cc:2229] T 05f9a54a02104f4c98caab7fbdf39e74 P 8188e2acd6ea4edb866fb2c07959d533 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:21:26.891526 12631 tablet_replica.cc:330] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1: stopping tablet replica
I20260924 03:21:26.891589 12630 raft_consensus.cc:2258] T 05f9a54a02104f4c98caab7fbdf39e74 P 8188e2acd6ea4edb866fb2c07959d533 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:21:26.891623 12629 raft_consensus.cc:2258] T 05f9a54a02104f4c98caab7fbdf39e74 P d0b7249d472b430ab4dfc3e312d4ea02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:21:26.891772 12631 raft_consensus.cc:2229] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:21:26.892619 12631 raft_consensus.cc:2258] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:21:26.893157 12629 ts_tablet_manager.cc:1905] T 05f9a54a02104f4c98caab7fbdf39e74 P d0b7249d472b430ab4dfc3e312d4ea02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:21:26.893160 12630 ts_tablet_manager.cc:1905] T 05f9a54a02104f4c98caab7fbdf39e74 P 8188e2acd6ea4edb866fb2c07959d533: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:21:26.895901 12054 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 03:21:26.897693 12631 ts_tablet_manager.cc:1905] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:21:26.901156 12629 ts_tablet_manager.cc:1918] T 05f9a54a02104f4c98caab7fbdf39e74 P d0b7249d472b430ab4dfc3e312d4ea02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:21:26.901156 12630 ts_tablet_manager.cc:1918] T 05f9a54a02104f4c98caab7fbdf39e74 P 8188e2acd6ea4edb866fb2c07959d533: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:21:26.901355 12629 log.cc:1196] T 05f9a54a02104f4c98caab7fbdf39e74 P d0b7249d472b430ab4dfc3e312d4ea02: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/05f9a54a02104f4c98caab7fbdf39e74
I20260924 03:21:26.901361 12630 log.cc:1196] T 05f9a54a02104f4c98caab7fbdf39e74 P 8188e2acd6ea4edb866fb2c07959d533: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-0/wal/wals/05f9a54a02104f4c98caab7fbdf39e74
I20260924 03:21:26.901784 12629 ts_tablet_manager.cc:1939] T 05f9a54a02104f4c98caab7fbdf39e74 P d0b7249d472b430ab4dfc3e312d4ea02: Deleting consensus metadata
I20260924 03:21:26.901803 12630 ts_tablet_manager.cc:1939] T 05f9a54a02104f4c98caab7fbdf39e74 P 8188e2acd6ea4edb866fb2c07959d533: Deleting consensus metadata
W20260924 03:21:26.903264 12054 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:21:26.903370 12043 catalog_manager.cc:4841] TS d0b7249d472b430ab4dfc3e312d4ea02 (127.10.12.131:35321): tablet 05f9a54a02104f4c98caab7fbdf39e74 (table test-table [id=76cd403927a1431c87d29a764f4a8676]) successfully deleted
I20260924 03:21:26.903731 12041 catalog_manager.cc:4841] TS 8188e2acd6ea4edb866fb2c07959d533 (127.10.12.129:38037): tablet 05f9a54a02104f4c98caab7fbdf39e74 (table test-table [id=76cd403927a1431c87d29a764f4a8676]) successfully deleted
I20260924 03:21:26.907308 12631 ts_tablet_manager.cc:1918] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:21:26.907636 12631 log.cc:1196] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/05f9a54a02104f4c98caab7fbdf39e74
I20260924 03:21:26.908735 12631 ts_tablet_manager.cc:1939] T 05f9a54a02104f4c98caab7fbdf39e74 P 61d88c04b6344d54a3aadac52a4b4ca1: Deleting consensus metadata
I20260924 03:21:26.915236 12471 tablet_service.cc:1467] Processing CreateTablet for tablet b5e378d4ecdc40c1bfa8a95e30abab22 (DEFAULT_TABLE table=test-table [id=1f3a7339cc6c4855957de4baeb3fab1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:26.915604 12042 catalog_manager.cc:4841] TS 61d88c04b6344d54a3aadac52a4b4ca1 (127.10.12.130:42369): tablet 05f9a54a02104f4c98caab7fbdf39e74 (table test-table [id=76cd403927a1431c87d29a764f4a8676]) successfully deleted
I20260924 03:21:26.916723 12332 tablet_service.cc:1467] Processing CreateTablet for tablet b5e378d4ecdc40c1bfa8a95e30abab22 (DEFAULT_TABLE table=test-table [id=1f3a7339cc6c4855957de4baeb3fab1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:26.917160 12193 tablet_service.cc:1467] Processing CreateTablet for tablet b5e378d4ecdc40c1bfa8a95e30abab22 (DEFAULT_TABLE table=test-table [id=1f3a7339cc6c4855957de4baeb3fab1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:26.917450 12332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5e378d4ecdc40c1bfa8a95e30abab22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:26.917951 12193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5e378d4ecdc40c1bfa8a95e30abab22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:26.919605 12471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5e378d4ecdc40c1bfa8a95e30abab22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:26.927240 12641 tablet_bootstrap.cc:492] T b5e378d4ecdc40c1bfa8a95e30abab22 P 8188e2acd6ea4edb866fb2c07959d533: Bootstrap starting.
I20260924 03:21:26.928136 12642 tablet_bootstrap.cc:492] T b5e378d4ecdc40c1bfa8a95e30abab22 P 61d88c04b6344d54a3aadac52a4b4ca1: Bootstrap starting.
I20260924 03:21:26.929479 12641 tablet_bootstrap.cc:654] T b5e378d4ecdc40c1bfa8a95e30abab22 P 8188e2acd6ea4edb866fb2c07959d533: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:26.930495 12642 tablet_bootstrap.cc:654] T b5e378d4ecdc40c1bfa8a95e30abab22 P 61d88c04b6344d54a3aadac52a4b4ca1: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:26.933318 12642 tablet_bootstrap.cc:492] T b5e378d4ecdc40c1bfa8a95e30abab22 P 61d88c04b6344d54a3aadac52a4b4ca1: No bootstrap required, opened a new log
I20260924 03:21:26.933732 12642 ts_tablet_manager.cc:1397] T b5e378d4ecdc40c1bfa8a95e30abab22 P 61d88c04b6344d54a3aadac52a4b4ca1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260924 03:21:26.934787 12641 tablet_bootstrap.cc:492] T b5e378d4ecdc40c1bfa8a95e30abab22 P 8188e2acd6ea4edb866fb2c07959d533: No bootstrap required, opened a new log
I20260924 03:21:26.935004 12641 ts_tablet_manager.cc:1397] T b5e378d4ecdc40c1bfa8a95e30abab22 P 8188e2acd6ea4edb866fb2c07959d533: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 03:21:26.935511 12640 tablet_bootstrap.cc:492] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02: Bootstrap starting.
I20260924 03:21:26.934743 12642 raft_consensus.cc:348] T b5e378d4ecdc40c1bfa8a95e30abab22 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:26.936167 12642 raft_consensus.cc:374] T b5e378d4ecdc40c1bfa8a95e30abab22 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:26.936270 12642 raft_consensus.cc:729] T b5e378d4ecdc40c1bfa8a95e30abab22 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d88c04b6344d54a3aadac52a4b4ca1, State: Initialized, Role: FOLLOWER
I20260924 03:21:26.936607 12642 consensus_queue.cc:260] T b5e378d4ecdc40c1bfa8a95e30abab22 P 61d88c04b6344d54a3aadac52a4b4ca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:26.937282 12642 ts_tablet_manager.cc:1428] T b5e378d4ecdc40c1bfa8a95e30abab22 P 61d88c04b6344d54a3aadac52a4b4ca1: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260924 03:21:26.937107 12641 raft_consensus.cc:348] T b5e378d4ecdc40c1bfa8a95e30abab22 P 8188e2acd6ea4edb866fb2c07959d533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:26.937482 12641 raft_consensus.cc:374] T b5e378d4ecdc40c1bfa8a95e30abab22 P 8188e2acd6ea4edb866fb2c07959d533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:26.937620 12641 raft_consensus.cc:729] T b5e378d4ecdc40c1bfa8a95e30abab22 P 8188e2acd6ea4edb866fb2c07959d533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8188e2acd6ea4edb866fb2c07959d533, State: Initialized, Role: FOLLOWER
I20260924 03:21:26.937700 12640 tablet_bootstrap.cc:654] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:26.937968 12641 consensus_queue.cc:260] T b5e378d4ecdc40c1bfa8a95e30abab22 P 8188e2acd6ea4edb866fb2c07959d533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:26.942317 12641 ts_tablet_manager.cc:1428] T b5e378d4ecdc40c1bfa8a95e30abab22 P 8188e2acd6ea4edb866fb2c07959d533: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20260924 03:21:26.946528 12640 tablet_bootstrap.cc:492] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02: No bootstrap required, opened a new log
I20260924 03:21:26.946733 12640 ts_tablet_manager.cc:1397] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20260924 03:21:26.947547 12640 raft_consensus.cc:348] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:26.947820 12640 raft_consensus.cc:374] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:26.947902 12640 raft_consensus.cc:729] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b7249d472b430ab4dfc3e312d4ea02, State: Initialized, Role: FOLLOWER
I20260924 03:21:26.948220 12640 consensus_queue.cc:260] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:26.949501 12640 ts_tablet_manager.cc:1428] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:21:27.059921 12612 raft_consensus.cc:482] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:21:27.060302 12612 raft_consensus.cc:504] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:27.061303 12612 leader_election.cc:288] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8188e2acd6ea4edb866fb2c07959d533 (127.10.12.129:38037), 61d88c04b6344d54a3aadac52a4b4ca1 (127.10.12.130:42369)
I20260924 03:21:27.062160 12356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5e378d4ecdc40c1bfa8a95e30abab22" candidate_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" is_pre_election: true
I20260924 03:21:27.062148 12212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5e378d4ecdc40c1bfa8a95e30abab22" candidate_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8188e2acd6ea4edb866fb2c07959d533" is_pre_election: true
I20260924 03:21:27.062465 12212 raft_consensus.cc:2454] T b5e378d4ecdc40c1bfa8a95e30abab22 P 8188e2acd6ea4edb866fb2c07959d533 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0b7249d472b430ab4dfc3e312d4ea02 in term 0.
I20260924 03:21:27.062480 12356 raft_consensus.cc:2454] T b5e378d4ecdc40c1bfa8a95e30abab22 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0b7249d472b430ab4dfc3e312d4ea02 in term 0.
I20260924 03:21:27.063056 12424 leader_election.cc:302] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 61d88c04b6344d54a3aadac52a4b4ca1, 8188e2acd6ea4edb866fb2c07959d533, d0b7249d472b430ab4dfc3e312d4ea02; no voters:
I20260924 03:21:27.063355 12612 raft_consensus.cc:2781] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:21:27.063490 12612 raft_consensus.cc:482] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:21:27.063597 12612 raft_consensus.cc:3037] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:27.066478 12612 raft_consensus.cc:504] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:27.067219 12612 leader_election.cc:288] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02 [CANDIDATE]: Term 1 election: Requested vote from peers 8188e2acd6ea4edb866fb2c07959d533 (127.10.12.129:38037), 61d88c04b6344d54a3aadac52a4b4ca1 (127.10.12.130:42369)
I20260924 03:21:27.068179 12212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5e378d4ecdc40c1bfa8a95e30abab22" candidate_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8188e2acd6ea4edb866fb2c07959d533"
I20260924 03:21:27.068480 12212 raft_consensus.cc:3037] T b5e378d4ecdc40c1bfa8a95e30abab22 P 8188e2acd6ea4edb866fb2c07959d533 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:27.068599 12356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5e378d4ecdc40c1bfa8a95e30abab22" candidate_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d88c04b6344d54a3aadac52a4b4ca1"
I20260924 03:21:27.068946 12356 raft_consensus.cc:3037] T b5e378d4ecdc40c1bfa8a95e30abab22 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:27.071095 12212 raft_consensus.cc:2454] T b5e378d4ecdc40c1bfa8a95e30abab22 P 8188e2acd6ea4edb866fb2c07959d533 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0b7249d472b430ab4dfc3e312d4ea02 in term 1.
I20260924 03:21:27.071758 12423 leader_election.cc:302] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8188e2acd6ea4edb866fb2c07959d533, d0b7249d472b430ab4dfc3e312d4ea02; no voters:
I20260924 03:21:27.072208 12612 raft_consensus.cc:2781] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:27.072557 12356 raft_consensus.cc:2454] T b5e378d4ecdc40c1bfa8a95e30abab22 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0b7249d472b430ab4dfc3e312d4ea02 in term 1.
I20260924 03:21:27.072724 12612 raft_consensus.cc:686] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02 [term 1 LEADER]: Becoming Leader. State: Replica: d0b7249d472b430ab4dfc3e312d4ea02, State: Running, Role: LEADER
I20260924 03:21:27.073081 12612 consensus_queue.cc:237] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } }
I20260924 03:21:27.076766 12054 catalog_manager.cc:5495] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0b7249d472b430ab4dfc3e312d4ea02 (127.10.12.131). New cstate: current_term: 1 leader_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0b7249d472b430ab4dfc3e312d4ea02" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 35321 } health_report { overall_health: HEALTHY } } }
I20260924 03:21:27.118844 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:27.113645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44294 (local address 127.10.12.131:35321)
0924 03:21:27.113865 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:27.113871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:27.113892 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:21:27.113973 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:27.113979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:27.114064 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:27.114141 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:27.114519 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:27.115320 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:27.116894 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:27.117363 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:27.117486 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:27.117978 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:27.118139 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:27.118218 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:27.118375 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 03:21:27.118476 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 03:21:27.134374 12212 raft_consensus.cc:1261] T b5e378d4ecdc40c1bfa8a95e30abab22 P 8188e2acd6ea4edb866fb2c07959d533 [term 1 FOLLOWER]: Refusing update from remote peer d0b7249d472b430ab4dfc3e312d4ea02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:21:27.135159 12617 consensus_queue.cc:1035] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8188e2acd6ea4edb866fb2c07959d533" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:21:27.142695 12354 raft_consensus.cc:1261] T b5e378d4ecdc40c1bfa8a95e30abab22 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 1 FOLLOWER]: Refusing update from remote peer d0b7249d472b430ab4dfc3e312d4ea02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:21:27.153396 12597 consensus_queue.cc:1035] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61d88c04b6344d54a3aadac52a4b4ca1" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 03:21:27.186573 12610 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 03:21:28.791568 12054 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:21:28.791846 12054 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:21:28.796801 12054 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63f05b4c617846e9970318185e930366: Sending DeleteTablet for 3 replicas of tablet b5e378d4ecdc40c1bfa8a95e30abab22
I20260924 03:21:28.797827 12332 tablet_service.cc:1514] Processing DeleteTablet for tablet b5e378d4ecdc40c1bfa8a95e30abab22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:21:28 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:40830
I20260924 03:21:28.797881 12193 tablet_service.cc:1514] Processing DeleteTablet for tablet b5e378d4ecdc40c1bfa8a95e30abab22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:21:28 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:50114
I20260924 03:21:28.798489 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 12127
I20260924 03:21:28.798545 12470 tablet_service.cc:1514] Processing DeleteTablet for tablet b5e378d4ecdc40c1bfa8a95e30abab22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:21:28 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:44256
I20260924 03:21:28.798741 12693 tablet_replica.cc:330] T b5e378d4ecdc40c1bfa8a95e30abab22 P 61d88c04b6344d54a3aadac52a4b4ca1: stopping tablet replica
I20260924 03:21:28.798799 12694 tablet_replica.cc:330] T b5e378d4ecdc40c1bfa8a95e30abab22 P 8188e2acd6ea4edb866fb2c07959d533: stopping tablet replica
I20260924 03:21:28.798942 12693 raft_consensus.cc:2229] T b5e378d4ecdc40c1bfa8a95e30abab22 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:21:28.798997 12694 raft_consensus.cc:2229] T b5e378d4ecdc40c1bfa8a95e30abab22 P 8188e2acd6ea4edb866fb2c07959d533 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:21:28.799197 12693 raft_consensus.cc:2258] T b5e378d4ecdc40c1bfa8a95e30abab22 P 61d88c04b6344d54a3aadac52a4b4ca1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:21:28.799242 12694 raft_consensus.cc:2258] T b5e378d4ecdc40c1bfa8a95e30abab22 P 8188e2acd6ea4edb866fb2c07959d533 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:21:28.800874 12693 ts_tablet_manager.cc:1905] T b5e378d4ecdc40c1bfa8a95e30abab22 P 61d88c04b6344d54a3aadac52a4b4ca1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:21:28.802021 12694 ts_tablet_manager.cc:1905] T b5e378d4ecdc40c1bfa8a95e30abab22 P 8188e2acd6ea4edb866fb2c07959d533: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:21:28.808490 12695 tablet_replica.cc:330] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02: stopping tablet replica
I20260924 03:21:28.808758 12695 raft_consensus.cc:2229] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:21:28.809177 12695 raft_consensus.cc:2258] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:21:28.809392 12693 ts_tablet_manager.cc:1918] T b5e378d4ecdc40c1bfa8a95e30abab22 P 61d88c04b6344d54a3aadac52a4b4ca1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:21:28.809559 12693 log.cc:1196] T b5e378d4ecdc40c1bfa8a95e30abab22 P 61d88c04b6344d54a3aadac52a4b4ca1: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/b5e378d4ecdc40c1bfa8a95e30abab22
I20260924 03:21:28.809964 12693 ts_tablet_manager.cc:1939] T b5e378d4ecdc40c1bfa8a95e30abab22 P 61d88c04b6344d54a3aadac52a4b4ca1: Deleting consensus metadata
I20260924 03:21:28.811013 12042 catalog_manager.cc:4841] TS 61d88c04b6344d54a3aadac52a4b4ca1 (127.10.12.130:42369): tablet b5e378d4ecdc40c1bfa8a95e30abab22 (table test-table [id=1f3a7339cc6c4855957de4baeb3fab1c]) successfully deleted
I20260924 03:21:28.811129 12694 ts_tablet_manager.cc:1918] T b5e378d4ecdc40c1bfa8a95e30abab22 P 8188e2acd6ea4edb866fb2c07959d533: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:21:28.811278 12694 log.cc:1196] T b5e378d4ecdc40c1bfa8a95e30abab22 P 8188e2acd6ea4edb866fb2c07959d533: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-0/wal/wals/b5e378d4ecdc40c1bfa8a95e30abab22
I20260924 03:21:28.811728 12694 ts_tablet_manager.cc:1939] T b5e378d4ecdc40c1bfa8a95e30abab22 P 8188e2acd6ea4edb866fb2c07959d533: Deleting consensus metadata
I20260924 03:21:28.812593 12695 ts_tablet_manager.cc:1905] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:21:28.812950 12041 catalog_manager.cc:4841] TS 8188e2acd6ea4edb866fb2c07959d533 (127.10.12.129:38037): tablet b5e378d4ecdc40c1bfa8a95e30abab22 (table test-table [id=1f3a7339cc6c4855957de4baeb3fab1c]) successfully deleted
I20260924 03:21:28.817890 12695 ts_tablet_manager.cc:1918] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:21:28.818030 12695 log.cc:1196] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/b5e378d4ecdc40c1bfa8a95e30abab22
I20260924 03:21:28.818477 12695 ts_tablet_manager.cc:1939] T b5e378d4ecdc40c1bfa8a95e30abab22 P d0b7249d472b430ab4dfc3e312d4ea02: Deleting consensus metadata
I20260924 03:21:28.819628 12043 catalog_manager.cc:4841] TS d0b7249d472b430ab4dfc3e312d4ea02 (127.10.12.131:35321): tablet b5e378d4ecdc40c1bfa8a95e30abab22 (table test-table [id=1f3a7339cc6c4855957de4baeb3fab1c]) successfully deleted
Sep 24 03:21:28 dist-test-slave-00qs krb5kdc[12011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220086, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.12.129@KRBTEST.COM
I20260924 03:21:28.835220 12697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:28.800141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50124 (local address 127.10.12.129:38037)
0924 03:21:28.813173 (+ 13032us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:28.813181 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:28.813682 (+ 501us) server_negotiation.cc:407] Connection header received
0924 03:21:28.813859 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:28.813863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:28.813928 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:28.814028 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:28.815264 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:28.816084 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:28.817394 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:28.817802 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:28.817920 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:28.828754 (+ 10834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:28.828773 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:28.828791 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:28.828836 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:28.831835 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:28.832610 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:28.832618 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:28.832621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:28.833192 (+ 571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:28.833601 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:28.833605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:28.833607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:28.833977 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:28.834289 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:28.834595 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 03:21:28.834856 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12867,"thread_start_us":12734,"threads_started":1}
W20260924 03:21:28.836064 12258 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50124
W20260924 03:21:28.836843 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 12127: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:21:28.837057 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 12127
I20260924 03:21:28.851353 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 12267
Sep 24 03:21:28 dist-test-slave-00qs krb5kdc[12011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220086, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.12.130@KRBTEST.COM
I20260924 03:21:28.868388 12698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:28.852249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40844 (local address 127.10.12.130:42369)
0924 03:21:28.852754 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:28.852759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:28.852776 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:21:28.852834 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:28.852838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:28.852899 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:28.852995 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:28.854365 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:28.855164 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:28.856612 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:28.857087 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:28.857206 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:28.861842 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:28.861860 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:28.861878 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:28.861922 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:28.865076 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:28.865909 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:28.865917 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:28.865920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:28.866172 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:28.866663 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:28.866667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:28.866669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:28.867150 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:28.867425 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:28.867808 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 03:21:28.868080 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":273,"threads_started":1}
W20260924 03:21:28.869249 12397 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40844
W20260924 03:21:28.870018 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 12267: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:21:28.870155 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 12267
I20260924 03:21:28.889582 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 12406
Sep 24 03:21:28 dist-test-slave-00qs krb5kdc[12011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220086, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.12.131@KRBTEST.COM
I20260924 03:21:28.909195 12699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:28.890823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44310 (local address 127.10.12.131:35321)
0924 03:21:28.891570 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:28.891577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:28.891600 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:21:28.891676 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:28.891682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:28.891760 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:28.891900 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:28.893529 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:28.894492 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:28.896152 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:28.896932 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:28.897117 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:28.901951 (+ 4834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:28.901971 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:28.901988 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:28.902036 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:28.905157 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:28.906422 (+ 1265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:28.906431 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:28.906434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:28.906610 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:28.907311 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:28.907315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:28.907318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:28.907726 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:28.908070 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:28.908595 (+ 525us) server_negotiation.cc:299] Negotiation successful
0924 03:21:28.908832 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":361,"threads_started":1}
W20260924 03:21:28.909986 12536 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44310
W20260924 03:21:28.910665 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 12406: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:21:28.910934 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 12406
I20260924 03:21:28.931867 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 12024
I20260924 03:21:28.933185 12096 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:29.051685 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 12024
2026-09-24T03:21:29Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8360 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:21:29 dist-test-slave-00qs krb5kdc[12703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:21:29 dist-test-slave-00qs krb5kdc[12703](info): set up 2 sockets
Sep 24 03:21:29 dist-test-slave-00qs krb5kdc[12703](info): commencing operation
krb5kdc: starting...
W20260924 03:21:30.088366 10290 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:21:30 dist-test-slave-00qs krb5kdc[12703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:21:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:21:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:21:30.294323 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:39823
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:39823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:30.658381 12716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:30.658707 12716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:30.658835 12716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:30.666399 12716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:21:30.666534 12716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:30.666581 12716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:30.666657 12716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:21:30.666719 12716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:21:30.677394 12716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:39823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:39823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:30.678872 12716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:30.680292 12716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:30.696508 12721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:30.696478 12724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:30.696535 12722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:30.696740 12716 server_base.cc:1034] running on GCE node
I20260924 03:21:30.697849 12716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:30.699507 12716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:30.700919 12716 hybrid_clock.cc:648] HybridClock initialized: now 1790220090700824 us; error 41 us; skew 500 ppm
I20260924 03:21:30.701542 12716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:30 dist-test-slave-00qs krb5kdc[12703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220090, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:30.706682 12716 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:21:30.708779 12716 webserver.cc:460] Webserver started at http://127.10.12.190:33839/ using document root <none> and password file <none>
I20260924 03:21:30.709576 12716 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:30.709712 12716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:30.710045 12716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:30.713428 12716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "274d53d6c306446c993e07d4b7720fc4"
format_stamp: "Formatted at 2026-09-24 03:21:30 on dist-test-slave-00qs"
server_key: "710b1f4628299b23f44e8a65448fad87"
server_key_iv: "5d2438be973ce18e5b42c76a1dabc31f"
server_key_version: "encryptionkey@0"
I20260924 03:21:30.714270 12716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "274d53d6c306446c993e07d4b7720fc4"
format_stamp: "Formatted at 2026-09-24 03:21:30 on dist-test-slave-00qs"
server_key: "710b1f4628299b23f44e8a65448fad87"
server_key_iv: "5d2438be973ce18e5b42c76a1dabc31f"
server_key_version: "encryptionkey@0"
I20260924 03:21:30.720304 12716 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 03:21:30.724362 12731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:30.726075 12716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:21:30.726285 12716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "274d53d6c306446c993e07d4b7720fc4"
format_stamp: "Formatted at 2026-09-24 03:21:30 on dist-test-slave-00qs"
server_key: "710b1f4628299b23f44e8a65448fad87"
server_key_iv: "5d2438be973ce18e5b42c76a1dabc31f"
server_key_version: "encryptionkey@0"
I20260924 03:21:30.726496 12716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:30.744846 12716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:30.750229 12716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:30.750564 12716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:30.769851 12716 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:39823
I20260924 03:21:30.769888 12782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:39823 every 8 connection(s)
I20260924 03:21:30.771328 12716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:21:30.775529 12783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:30.777614 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 12716
I20260924 03:21:30.777851 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:21:30.778738 10290 external_mini_cluster.cc:1428] Setting key 5b21356c0203b109de64a04f6ea587ad
I20260924 03:21:30.789112 12783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 274d53d6c306446c993e07d4b7720fc4: Bootstrap starting.
I20260924 03:21:30.793048 12783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 274d53d6c306446c993e07d4b7720fc4: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:30.794400 12783 log.cc:824] T 00000000000000000000000000000000 P 274d53d6c306446c993e07d4b7720fc4: Log is configured to *not* fsync() on all Append() calls
Sep 24 03:21:30 dist-test-slave-00qs krb5kdc[12703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:30.797955 12783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 274d53d6c306446c993e07d4b7720fc4: No bootstrap required, opened a new log
I20260924 03:21:30.807107 12786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:30.781220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55020 (local address 127.10.12.190:39823)
0924 03:21:30.782142 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:30.782158 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:30.782203 (+ 45us) server_negotiation.cc:407] Connection header received
0924 03:21:30.784619 (+ 2416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:30.784664 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:30.785429 (+ 765us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:30.785996 (+ 567us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:30.788011 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:30.789466 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:30.791336 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:30.792271 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:30.792438 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:30.797359 (+ 4921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:30.797372 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:30.797394 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:30.797451 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:30.801637 (+ 4186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:30.802630 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:30.802642 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:30.802653 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:30.802889 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:30.803323 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:30.803329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:30.803336 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:30.804205 (+ 869us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:30.804528 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:30.804973 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 03:21:30.805488 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":465,"threads_started":1}
I20260924 03:21:30.808694 12783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 274d53d6c306446c993e07d4b7720fc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274d53d6c306446c993e07d4b7720fc4" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39823 } }
I20260924 03:21:30.809104 12783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 274d53d6c306446c993e07d4b7720fc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:30.809209 12783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 274d53d6c306446c993e07d4b7720fc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274d53d6c306446c993e07d4b7720fc4, State: Initialized, Role: FOLLOWER
I20260924 03:21:30.809787 12783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 274d53d6c306446c993e07d4b7720fc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274d53d6c306446c993e07d4b7720fc4" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39823 } }
I20260924 03:21:30.809998 12783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 274d53d6c306446c993e07d4b7720fc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:21:30.810140 12783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 274d53d6c306446c993e07d4b7720fc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:21:30.810356 12783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 274d53d6c306446c993e07d4b7720fc4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:30.812767 12783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 274d53d6c306446c993e07d4b7720fc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274d53d6c306446c993e07d4b7720fc4" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39823 } }
I20260924 03:21:30.813395 12783 leader_election.cc:302] T 00000000000000000000000000000000 P 274d53d6c306446c993e07d4b7720fc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 274d53d6c306446c993e07d4b7720fc4; no voters:
I20260924 03:21:30.814085 12783 leader_election.cc:288] T 00000000000000000000000000000000 P 274d53d6c306446c993e07d4b7720fc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:21:30.814229 12788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 274d53d6c306446c993e07d4b7720fc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:30.814966 12788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 274d53d6c306446c993e07d4b7720fc4 [term 1 LEADER]: Becoming Leader. State: Replica: 274d53d6c306446c993e07d4b7720fc4, State: Running, Role: LEADER
I20260924 03:21:30.815521 12788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 274d53d6c306446c993e07d4b7720fc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274d53d6c306446c993e07d4b7720fc4" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39823 } }
I20260924 03:21:30.816603 12783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 274d53d6c306446c993e07d4b7720fc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:21:30.818501 12789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 274d53d6c306446c993e07d4b7720fc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "274d53d6c306446c993e07d4b7720fc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274d53d6c306446c993e07d4b7720fc4" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39823 } } }
I20260924 03:21:30.818922 12789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 274d53d6c306446c993e07d4b7720fc4 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:30.819825 12790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 274d53d6c306446c993e07d4b7720fc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 274d53d6c306446c993e07d4b7720fc4. Latest consensus state: current_term: 1 leader_uuid: "274d53d6c306446c993e07d4b7720fc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274d53d6c306446c993e07d4b7720fc4" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39823 } } }
I20260924 03:21:30.820209 12790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 274d53d6c306446c993e07d4b7720fc4 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:30.821949 12795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:21:30.826961 12795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:21:30.837375 12795 catalog_manager.cc:1347] Generated new cluster ID: f79952ad52664f759f59b166578d55fc
I20260924 03:21:30.837512 12795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:21:30.850761 12795 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:21:30.852257 12795 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:21:30.865543 12795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 274d53d6c306446c993e07d4b7720fc4: Generated new TSK 0
I20260924 03:21:30.866501 12795 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
I20260924 03:21:30.920651 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:39823
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:31.259635 12809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:31.259972 12809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:31.260104 12809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:31.268009 12809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:31.268203 12809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:31.268369 12809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:21:31.278744 12809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:39823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:31.280145 12809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:31.281525 12809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:31.295990 12814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:31.295944 12815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:31.296031 12817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:31.296427 12809 server_base.cc:1034] running on GCE node
I20260924 03:21:31.297201 12809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:31.298153 12809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:31.299373 12809 hybrid_clock.cc:648] HybridClock initialized: now 1790220091299321 us; error 37 us; skew 500 ppm
I20260924 03:21:31.299695 12809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:31 dist-test-slave-00qs krb5kdc[12703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220091, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:31.303157 12809 init.cc:377] Logged in from keytab as kudu/127.10.12.129@KRBTEST.COM (short username kudu)
I20260924 03:21:31.304803 12809 webserver.cc:460] Webserver started at http://127.10.12.129:35635/ using document root <none> and password file <none>
I20260924 03:21:31.305545 12809 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:31.305655 12809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:31.305965 12809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:31.309118 12809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "e4e50d4a65a8423aaf427d705d442e8e"
format_stamp: "Formatted at 2026-09-24 03:21:31 on dist-test-slave-00qs"
server_key: "d0b2c6101e94085e251b16eea0047d96"
server_key_iv: "2bea1fcee75fb6171ebbbe318b96a533"
server_key_version: "encryptionkey@0"
I20260924 03:21:31.309942 12809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "e4e50d4a65a8423aaf427d705d442e8e"
format_stamp: "Formatted at 2026-09-24 03:21:31 on dist-test-slave-00qs"
server_key: "d0b2c6101e94085e251b16eea0047d96"
server_key_iv: "2bea1fcee75fb6171ebbbe318b96a533"
server_key_version: "encryptionkey@0"
I20260924 03:21:31.315584 12809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:21:31.319492 12824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:31.321080 12809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 03:21:31.321274 12809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "e4e50d4a65a8423aaf427d705d442e8e"
format_stamp: "Formatted at 2026-09-24 03:21:31 on dist-test-slave-00qs"
server_key: "d0b2c6101e94085e251b16eea0047d96"
server_key_iv: "2bea1fcee75fb6171ebbbe318b96a533"
server_key_version: "encryptionkey@0"
I20260924 03:21:31.321494 12809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:31.345616 12809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:31.350211 12809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:31.350507 12809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:31.351526 12809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:21:31.352968 12809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:31.353045 12809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:31.353152 12809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:31.353207 12809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:31.384146 12809 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:43447
I20260924 03:21:31.384184 12936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:43447 every 8 connection(s)
I20260924 03:21:31.385492 12809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:21:31.392266 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 12809
I20260924 03:21:31.392505 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:21:31.393126 10290 external_mini_cluster.cc:1428] Setting key fa98ec3a34be22740f313cc48a2e57bc
Sep 24 03:21:31 dist-test-slave-00qs krb5kdc[12703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220091, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:31.407629 12939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:31.388470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:40043 (local address 127.10.12.190:39823)
0924 03:21:31.388985 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:31.388992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:31.390019 (+ 1027us) server_negotiation.cc:407] Connection header received
0924 03:21:31.391931 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:31.391938 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:31.392059 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:31.392188 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:31.394057 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:31.394904 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:31.396682 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:31.397242 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:31.397370 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:31.401057 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:31.401064 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:31.401069 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:31.401109 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:31.404294 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:31.405122 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:31.405128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:31.405131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:31.405277 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:31.405693 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:31.405698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:31.405701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:31.406059 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:31.406199 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:31.407022 (+ 823us) server_negotiation.cc:299] Negotiation successful
0924 03:21:31.407259 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":211,"threads_started":1}
I20260924 03:21:31.408603 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:31.389098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39823 (local address 127.10.12.129:40043)
0924 03:21:31.389848 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:31.389896 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:31.391602 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:31.392442 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:31.392454 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:31.393102 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:31.393864 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:31.393894 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:31.395080 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:31.395086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:31.396522 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:31.396537 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:31.397503 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:31.397508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:31.397722 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:31.398467 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:31.398491 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:31.400803 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:31.404499 (+ 3696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:31.404507 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:31.404524 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:31.405006 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:31.405380 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:31.405383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:31.405385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:31.405584 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:31.406209 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:31.406216 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:31.406622 (+ 406us) client_negotiation.cc:769] Sending connection context
0924 03:21:31.406892 (+ 270us) client_negotiation.cc:240] Negotiation successful
0924 03:21:31.407331 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":283,"threads_started":1}
I20260924 03:21:31.410156 12937 heartbeater.cc:347] Connected to a master server at 127.10.12.190:39823
I20260924 03:21:31.410529 12937 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.12.130@KRBTEST.COM; defaulting to no policy
I20260924 03:21:31.411404 12937 heartbeater.cc:511] Master 127.10.12.190:39823 requested a full tablet report, sending...
I20260924 03:21:31.413846 12748 ts_manager.cc:194] Registered new tserver with Master: e4e50d4a65a8423aaf427d705d442e8e (127.10.12.129:43447)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.130@KRBTEST.COM" created.
I20260924 03:21:31.415416 12748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.129@KRBTEST.COM'} at 127.10.12.129:40043
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
I20260924 03:21:31.441066 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:39823
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:31.791905 12943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:31.792243 12943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:31.792364 12943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:31.799803 12943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:31.800063 12943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:31.800241 12943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:21:31.810520 12943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:39823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:31.812543 12943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:31.814191 12943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:31.828722 12948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:31.828902 12949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:31.828931 12951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:31.830243 12943 server_base.cc:1034] running on GCE node
I20260924 03:21:31.831039 12943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:31.832244 12943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:31.833566 12943 hybrid_clock.cc:648] HybridClock initialized: now 1790220091833498 us; error 62 us; skew 500 ppm
I20260924 03:21:31.833992 12943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:31 dist-test-slave-00qs krb5kdc[12703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220091, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:31.837994 12943 init.cc:377] Logged in from keytab as kudu/127.10.12.130@KRBTEST.COM (short username kudu)
I20260924 03:21:31.839707 12943 webserver.cc:460] Webserver started at http://127.10.12.130:43817/ using document root <none> and password file <none>
I20260924 03:21:31.840507 12943 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:31.840621 12943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:31.840919 12943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:31.844134 12943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "3d50400c9cba49ad8e6cfa48275d4004"
format_stamp: "Formatted at 2026-09-24 03:21:31 on dist-test-slave-00qs"
server_key: "5c1ef17f7e883867eb20f03e34e6f06a"
server_key_iv: "0d390a78aca22204f6cdf3950fe25103"
server_key_version: "encryptionkey@0"
I20260924 03:21:31.845017 12943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "3d50400c9cba49ad8e6cfa48275d4004"
format_stamp: "Formatted at 2026-09-24 03:21:31 on dist-test-slave-00qs"
server_key: "5c1ef17f7e883867eb20f03e34e6f06a"
server_key_iv: "0d390a78aca22204f6cdf3950fe25103"
server_key_version: "encryptionkey@0"
I20260924 03:21:31.851606 12943 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 03:21:31.856774 12958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:31.858733 12943 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:21:31.858945 12943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "3d50400c9cba49ad8e6cfa48275d4004"
format_stamp: "Formatted at 2026-09-24 03:21:31 on dist-test-slave-00qs"
server_key: "5c1ef17f7e883867eb20f03e34e6f06a"
server_key_iv: "0d390a78aca22204f6cdf3950fe25103"
server_key_version: "encryptionkey@0"
I20260924 03:21:31.859166 12943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:31.889503 12943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:31.894344 12943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:31.894626 12943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:31.895702 12943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:21:31.897289 12943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:31.897373 12943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:31.897496 12943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:31.897558 12943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:31.929687 12943 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:33911
I20260924 03:21:31.929729 13070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:33911 every 8 connection(s)
I20260924 03:21:31.931159 12943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:21:31.941159 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 12943
I20260924 03:21:31.941370 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:21:31.941915 10290 external_mini_cluster.cc:1428] Setting key 7634db5554a2124dc10ada141eccda40
Sep 24 03:21:31 dist-test-slave-00qs krb5kdc[12703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220091, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:31.954991 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:31.934815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:52837 (local address 127.10.12.190:39823)
0924 03:21:31.935457 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:31.935465 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:31.936748 (+ 1283us) server_negotiation.cc:407] Connection header received
0924 03:21:31.939545 (+ 2797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:31.939552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:31.939657 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:31.939796 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:31.942107 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:31.942965 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:31.944843 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:31.945287 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:31.945418 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:31.949077 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:31.949083 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:31.949087 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:31.949123 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:31.951644 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:31.952538 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:31.952543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:31.952545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:31.952678 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:31.953139 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:31.953144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:31.953146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:31.953452 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:31.953589 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:31.954372 (+ 783us) server_negotiation.cc:299] Negotiation successful
0924 03:21:31.954626 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":289,"threads_started":1}
I20260924 03:21:31.956132 13074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:31.935480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39823 (local address 127.10.12.130:52837)
0924 03:21:31.936553 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:31.936617 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:31.939119 (+ 2502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:31.940232 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:31.940245 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:31.941073 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:31.941874 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:31.941897 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:31.943165 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:31.943170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:31.944681 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:31.944702 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:31.945522 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:31.945530 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:31.945760 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:31.946433 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:31.946456 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:31.948776 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:31.951827 (+ 3051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:31.951837 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:31.951853 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:31.952394 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:31.952791 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:31.952796 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:31.952800 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:31.953014 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:31.953582 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:31.953592 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:31.953980 (+ 388us) client_negotiation.cc:769] Sending connection context
0924 03:21:31.954232 (+ 252us) client_negotiation.cc:240] Negotiation successful
0924 03:21:31.954811 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":499,"threads_started":1}
WARNING: no policy specified for kudu/127.10.12.131@KRBTEST.COM; defaulting to no policy
I20260924 03:21:31.958127 13071 heartbeater.cc:347] Connected to a master server at 127.10.12.190:39823
I20260924 03:21:31.958536 13071 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:31.959322 13071 heartbeater.cc:511] Master 127.10.12.190:39823 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.131@KRBTEST.COM" created.
I20260924 03:21:31.961328 12748 ts_manager.cc:194] Registered new tserver with Master: 3d50400c9cba49ad8e6cfa48275d4004 (127.10.12.130:33911)
I20260924 03:21:31.962227 12748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.130@KRBTEST.COM'} at 127.10.12.130:52837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
I20260924 03:21:31.988420 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:39823
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:32.323812 13077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:32.324165 13077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:32.324303 13077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:32.332376 13077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:32.332527 13077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:32.332682 13077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:21:32.343429 13077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:39823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:32.344872 13077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:32.346686 13077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:32.360395 13083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:32.360488 13085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:32.360607 13082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:32.361359 13077 server_base.cc:1034] running on GCE node
I20260924 03:21:32.362003 13077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:32.363018 13077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:32.364248 13077 hybrid_clock.cc:648] HybridClock initialized: now 1790220092364190 us; error 47 us; skew 500 ppm
I20260924 03:21:32.364600 13077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:32 dist-test-slave-00qs krb5kdc[12703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220092, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:32.368218 13077 init.cc:377] Logged in from keytab as kudu/127.10.12.131@KRBTEST.COM (short username kudu)
I20260924 03:21:32.369877 13077 webserver.cc:460] Webserver started at http://127.10.12.131:33209/ using document root <none> and password file <none>
I20260924 03:21:32.370621 13077 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:32.370764 13077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:32.371086 13077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:32.374397 13077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "140603f63f8649a1a262a46bcae7cbae"
format_stamp: "Formatted at 2026-09-24 03:21:32 on dist-test-slave-00qs"
server_key: "2d940195d759c41cb85afdc963b118b8"
server_key_iv: "b62b24dd6245008db6146657e7e55e7d"
server_key_version: "encryptionkey@0"
I20260924 03:21:32.375257 13077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "140603f63f8649a1a262a46bcae7cbae"
format_stamp: "Formatted at 2026-09-24 03:21:32 on dist-test-slave-00qs"
server_key: "2d940195d759c41cb85afdc963b118b8"
server_key_iv: "b62b24dd6245008db6146657e7e55e7d"
server_key_version: "encryptionkey@0"
I20260924 03:21:32.380728 13077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 03:21:32.384521 13092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:32.386170 13077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260924 03:21:32.386346 13077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "140603f63f8649a1a262a46bcae7cbae"
format_stamp: "Formatted at 2026-09-24 03:21:32 on dist-test-slave-00qs"
server_key: "2d940195d759c41cb85afdc963b118b8"
server_key_iv: "b62b24dd6245008db6146657e7e55e7d"
server_key_version: "encryptionkey@0"
I20260924 03:21:32.386560 13077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:32.407248 13077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:32.411841 13077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:32.412206 13077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:32.413232 13077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:21:32.414736 13077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:32.414935 13077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:32.415092 13077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:32.415150 13077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:32.418921 12937 heartbeater.cc:503] Master 127.10.12.190:39823 was elected leader, sending a full tablet report...
I20260924 03:21:32.449608 13077 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:38289
I20260924 03:21:32.449647 13204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:38289 every 8 connection(s)
I20260924 03:21:32.450930 13077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:21:32.458770 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 13077
I20260924 03:21:32.458961 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:21:32.459355 10290 external_mini_cluster.cc:1428] Setting key 07be2bbffd73ee369270d7e3499b3292
Sep 24 03:21:32 dist-test-slave-00qs krb5kdc[12703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220092, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:32.475032 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:32.454073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:38355 (local address 127.10.12.190:39823)
0924 03:21:32.454421 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:32.454440 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:32.455625 (+ 1185us) server_negotiation.cc:407] Connection header received
0924 03:21:32.457650 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:32.457657 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:32.457768 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:32.457860 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:32.459603 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:32.460518 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:32.462773 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:32.463289 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:32.463430 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:32.468099 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:32.468105 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:32.468109 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:32.468150 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:32.470846 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:32.472209 (+ 1363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:32.472214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:32.472216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:32.472368 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:32.472819 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:32.472822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:32.472824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:32.473173 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:32.473332 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:32.474462 (+ 1130us) server_negotiation.cc:299] Negotiation successful
0924 03:21:32.474713 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260924 03:21:32.476099 13207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:32.454266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39823 (local address 127.10.12.131:38355)
0924 03:21:32.455449 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:32.455491 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:32.457315 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:32.458136 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:32.458145 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:32.458724 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:32.459408 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:32.459434 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:32.460696 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:32.460702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:32.462590 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:32.462611 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:32.463543 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:32.463548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:32.463803 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:32.464736 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:32.464781 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:32.467758 (+ 2977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:32.471145 (+ 3387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:32.471158 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:32.471179 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:32.472067 (+ 888us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:32.472486 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:32.472489 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:32.472492 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:32.472714 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:32.473344 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:32.473354 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:32.473880 (+ 526us) client_negotiation.cc:769] Sending connection context
0924 03:21:32.474227 (+ 347us) client_negotiation.cc:240] Negotiation successful
0924 03:21:32.474761 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"thread_start_us":515,"threads_started":1}
I20260924 03:21:32.477569 13205 heartbeater.cc:347] Connected to a master server at 127.10.12.190:39823
I20260924 03:21:32.477888 13205 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:32.478678 13205 heartbeater.cc:511] Master 127.10.12.190:39823 requested a full tablet report, sending...
I20260924 03:21:32.480279 12748 ts_manager.cc:194] Registered new tserver with Master: 140603f63f8649a1a262a46bcae7cbae (127.10.12.131:38289)
I20260924 03:21:32.481061 12748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.131@KRBTEST.COM'} at 127.10.12.131:38355
I20260924 03:21:32.489149 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 03:21:32.518743 13073 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:21:32.516384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55026 (local address 127.10.12.190:39823)
0924 03:21:32.516627 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:32.516634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:32.516924 (+ 290us) server_negotiation.cc:407] Connection header received
0924 03:21:32.517104 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:32.517110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:32.517177 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:32.517270 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:32.518469 (+ 1199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55026: BlockingRecv error: recv got EOF from 127.0.0.1:55026 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 03:21:32.521842 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 12809
W20260924 03:21:32.524571 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 12809: Not authorized: Client connection negotiation failed: client connection to 127.10.12.129:43447: server requires authentication, but client does not have Kerberos credentials available
I20260924 03:21:32.524734 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 12809
I20260924 03:21:32.538056 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 12943
W20260924 03:21:32.541204 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 12943: Not authorized: Client connection negotiation failed: client connection to 127.10.12.130:33911: server requires authentication, but client does not have Kerberos credentials available
I20260924 03:21:32.541368 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 12943
I20260924 03:21:32.553202 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 13077
W20260924 03:21:32.556676 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 13077: Not authorized: Client connection negotiation failed: client connection to 127.10.12.131:38289: server requires authentication, but client does not have Kerberos credentials available
I20260924 03:21:32.556892 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 13077
I20260924 03:21:32.568924 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 12716
I20260924 03:21:32.570184 12778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:32.674408 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 12716
2026-09-24T03:21:32Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3619 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:21:32 dist-test-slave-00qs krb5kdc[13222](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:21:32 dist-test-slave-00qs krb5kdc[13222](info): set up 2 sockets
Sep 24 03:21:32 dist-test-slave-00qs krb5kdc[13222](info): commencing operation
krb5kdc: starting...
W20260924 03:21:33.692721 10290 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:21:33 dist-test-slave-00qs krb5kdc[13222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:21:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:21:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:21:33.901170 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:38413
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:38465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:38413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:34.235234 13235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:34.235572 13235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:34.235688 13235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:34.243682 13235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:21:34.243831 13235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:34.243877 13235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:34.243978 13235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:21:34.244048 13235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:21:34.254436 13235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:38465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:38413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:38413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:34.255772 13235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:34.257113 13235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:34.269927 13240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:34.269948 13241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:34.270200 13235 server_base.cc:1034] running on GCE node
W20260924 03:21:34.269891 13243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:34.271597 13235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:34.273474 13235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:34.274741 13235 hybrid_clock.cc:648] HybridClock initialized: now 1790220094274691 us; error 44 us; skew 500 ppm
I20260924 03:21:34.275144 13235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:34 dist-test-slave-00qs krb5kdc[13222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220094, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:34.285772 13235 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:21:34.287474 13235 webserver.cc:460] Webserver started at http://127.10.12.190:36443/ using document root <none> and password file <none>
I20260924 03:21:34.288290 13235 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:34.288414 13235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:34.288727 13235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:34.291920 13235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "a3dc1f741a8d41259b30b85ca6a03c52"
format_stamp: "Formatted at 2026-09-24 03:21:34 on dist-test-slave-00qs"
server_key: "2dbb4f2a01ba12da0e2c64e59e24b80a"
server_key_iv: "03fd020d02afa8396602dc17a4bbd57f"
server_key_version: "encryptionkey@0"
I20260924 03:21:34.292833 13235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "a3dc1f741a8d41259b30b85ca6a03c52"
format_stamp: "Formatted at 2026-09-24 03:21:34 on dist-test-slave-00qs"
server_key: "2dbb4f2a01ba12da0e2c64e59e24b80a"
server_key_iv: "03fd020d02afa8396602dc17a4bbd57f"
server_key_version: "encryptionkey@0"
I20260924 03:21:34.298540 13235 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:21:34.302568 13250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:34.304244 13235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:21:34.304437 13235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "a3dc1f741a8d41259b30b85ca6a03c52"
format_stamp: "Formatted at 2026-09-24 03:21:34 on dist-test-slave-00qs"
server_key: "2dbb4f2a01ba12da0e2c64e59e24b80a"
server_key_iv: "03fd020d02afa8396602dc17a4bbd57f"
server_key_version: "encryptionkey@0"
I20260924 03:21:34.304639 13235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:34.322785 13235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:34.327286 13235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:34.327574 13235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:34.347112 13301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:38413 every 8 connection(s)
I20260924 03:21:34.347184 13235 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:38413
I20260924 03:21:34.348667 13235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:21:34.351354 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 13235
I20260924 03:21:34.351586 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:21:34.352392 10290 external_mini_cluster.cc:1428] Setting key 079165002b9038f024064ecfb40e9220
I20260924 03:21:34.353253 13302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 03:21:34 dist-test-slave-00qs krb5kdc[13222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:34.368475 13302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3dc1f741a8d41259b30b85ca6a03c52: Bootstrap starting.
I20260924 03:21:34.373556 13302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3dc1f741a8d41259b30b85ca6a03c52: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:34.375042 13302 log.cc:824] T 00000000000000000000000000000000 P a3dc1f741a8d41259b30b85ca6a03c52: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:34.377966 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:34.354310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45584 (local address 127.10.12.190:38413)
0924 03:21:34.355130 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:34.355142 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:34.355184 (+ 42us) server_negotiation.cc:407] Connection header received
0924 03:21:34.356793 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:34.356819 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:34.357267 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:34.357646 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:34.359853 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:34.361380 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:34.363110 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:34.363860 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:34.364071 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:34.369070 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:34.369078 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:34.369096 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:34.369144 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:34.372747 (+ 3603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:34.373960 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:34.373968 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:34.373976 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:34.374185 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:34.374647 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:34.374651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:34.374654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:34.375411 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:34.375728 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:34.376286 (+ 558us) server_negotiation.cc:299] Negotiation successful
0924 03:21:34.376829 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":364,"threads_started":1}
I20260924 03:21:34.378487 13302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3dc1f741a8d41259b30b85ca6a03c52: No bootstrap required, opened a new log
I20260924 03:21:34.381912 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:21:34.386386 13302 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3dc1f741a8d41259b30b85ca6a03c52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3dc1f741a8d41259b30b85ca6a03c52" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 38413 } }
I20260924 03:21:34.386930 13302 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3dc1f741a8d41259b30b85ca6a03c52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:34.387035 13302 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3dc1f741a8d41259b30b85ca6a03c52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3dc1f741a8d41259b30b85ca6a03c52, State: Initialized, Role: FOLLOWER
I20260924 03:21:34.387777 13302 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3dc1f741a8d41259b30b85ca6a03c52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3dc1f741a8d41259b30b85ca6a03c52" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 38413 } }
I20260924 03:21:34.388115 13302 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3dc1f741a8d41259b30b85ca6a03c52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:21:34.388257 13302 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3dc1f741a8d41259b30b85ca6a03c52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:21:34.388471 13302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3dc1f741a8d41259b30b85ca6a03c52 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:34.390767 13302 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3dc1f741a8d41259b30b85ca6a03c52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3dc1f741a8d41259b30b85ca6a03c52" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 38413 } }
I20260924 03:21:34.391358 13302 leader_election.cc:302] T 00000000000000000000000000000000 P a3dc1f741a8d41259b30b85ca6a03c52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3dc1f741a8d41259b30b85ca6a03c52; no voters:
I20260924 03:21:34.392035 13302 leader_election.cc:288] T 00000000000000000000000000000000 P a3dc1f741a8d41259b30b85ca6a03c52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:21:34.392143 13308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3dc1f741a8d41259b30b85ca6a03c52 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:34.393327 13308 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3dc1f741a8d41259b30b85ca6a03c52 [term 1 LEADER]: Becoming Leader. State: Replica: a3dc1f741a8d41259b30b85ca6a03c52, State: Running, Role: LEADER
I20260924 03:21:34.393944 13308 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3dc1f741a8d41259b30b85ca6a03c52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3dc1f741a8d41259b30b85ca6a03c52" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 38413 } }
I20260924 03:21:34.394506 13302 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3dc1f741a8d41259b30b85ca6a03c52 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:21:34.397306 13310 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3dc1f741a8d41259b30b85ca6a03c52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3dc1f741a8d41259b30b85ca6a03c52. Latest consensus state: current_term: 1 leader_uuid: "a3dc1f741a8d41259b30b85ca6a03c52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3dc1f741a8d41259b30b85ca6a03c52" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 38413 } } }
I20260924 03:21:34.397742 13310 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3dc1f741a8d41259b30b85ca6a03c52 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:34.400450 13309 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3dc1f741a8d41259b30b85ca6a03c52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3dc1f741a8d41259b30b85ca6a03c52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3dc1f741a8d41259b30b85ca6a03c52" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 38413 } } }
I20260924 03:21:34.401019 13309 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3dc1f741a8d41259b30b85ca6a03c52 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:34.404893 13315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:21:34.411078 13315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:21:34.422289 13315 catalog_manager.cc:1347] Generated new cluster ID: c7b18959030648c4bfb4cffbb4e1855e
I20260924 03:21:34.422431 13315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:21:34.433975 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:34.421859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45594 (local address 127.10.12.190:38413)
0924 03:21:34.422046 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:34.422053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:34.424723 (+ 2670us) server_negotiation.cc:407] Connection header received
0924 03:21:34.424928 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:34.424934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:34.425012 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:34.425130 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:34.426434 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:34.427559 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:34.429802 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:34.430481 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:34.430619 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:34.432068 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:34.432074 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:34.432079 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:21:34.433014 (+ 935us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:34.433081 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:21:34.433126 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:34.433240 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:34.433437 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 03:21:34.433558 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 03:21:34.439071 13315 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:21:34.441325 13315 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:21:34.451105 13315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3dc1f741a8d41259b30b85ca6a03c52: Generated new TSK 0
I20260924 03:21:34.452415 13315 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:21:34.464067 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 13235
I20260924 03:21:34.465545 13297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:34.557013 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 13235
2026-09-24T03:21:34Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1877 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:21:34 dist-test-slave-00qs krb5kdc[13336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:21:34 dist-test-slave-00qs krb5kdc[13336](info): set up 2 sockets
Sep 24 03:21:34 dist-test-slave-00qs krb5kdc[13336](info): commencing operation
krb5kdc: starting...
W20260924 03:21:35.584527 10290 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.007s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:21:35 dist-test-slave-00qs krb5kdc[13336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:21:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:21:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:21:35.792478 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:43619
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:43619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:36.131567 13349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:36.131901 13349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:36.132056 13349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:36.139588 13349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:21:36.139750 13349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:36.139797 13349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:36.139854 13349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:21:36.139915 13349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:21:36.150552 13349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:43619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:43619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:36.152030 13349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:36.153331 13349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:36.165618 13354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:36.165706 13357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:36.165647 13355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:36.166610 13349 server_base.cc:1034] running on GCE node
I20260924 03:21:36.167497 13349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:36.169188 13349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:36.170439 13349 hybrid_clock.cc:648] HybridClock initialized: now 1790220096170396 us; error 36 us; skew 500 ppm
I20260924 03:21:36.170902 13349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:36 dist-test-slave-00qs krb5kdc[13336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220096, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:36.174197 13349 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:21:36.176020 13349 webserver.cc:460] Webserver started at http://127.10.12.190:35031/ using document root <none> and password file <none>
I20260924 03:21:36.176766 13349 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:36.176899 13349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:36.177222 13349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:36.180480 13349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "4b6e172a2f5d4262a2994b1b5e478808"
format_stamp: "Formatted at 2026-09-24 03:21:36 on dist-test-slave-00qs"
server_key: "1f6cd1dd097d830ef925c6803d8275b7"
server_key_iv: "ac336bc3fb062261884ec3fbf4765ac0"
server_key_version: "encryptionkey@0"
I20260924 03:21:36.181324 13349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "4b6e172a2f5d4262a2994b1b5e478808"
format_stamp: "Formatted at 2026-09-24 03:21:36 on dist-test-slave-00qs"
server_key: "1f6cd1dd097d830ef925c6803d8275b7"
server_key_iv: "ac336bc3fb062261884ec3fbf4765ac0"
server_key_version: "encryptionkey@0"
I20260924 03:21:36.187780 13349 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 03:21:36.192162 13364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:36.193877 13349 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:21:36.194088 13349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "4b6e172a2f5d4262a2994b1b5e478808"
format_stamp: "Formatted at 2026-09-24 03:21:36 on dist-test-slave-00qs"
server_key: "1f6cd1dd097d830ef925c6803d8275b7"
server_key_iv: "ac336bc3fb062261884ec3fbf4765ac0"
server_key_version: "encryptionkey@0"
I20260924 03:21:36.194306 13349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:36.214210 13349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:36.218807 13349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:36.219098 13349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:36.239760 13349 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:43619
I20260924 03:21:36.239909 13415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:43619 every 8 connection(s)
I20260924 03:21:36.241255 13349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:21:36.242841 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 13349
I20260924 03:21:36.243067 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:21:36.243831 10290 external_mini_cluster.cc:1428] Setting key 3546fbf72357a924d30fecaa17a85f9d
I20260924 03:21:36.246387 13416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 03:21:36 dist-test-slave-00qs krb5kdc[13336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:36.261518 13416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b6e172a2f5d4262a2994b1b5e478808: Bootstrap starting.
I20260924 03:21:36.265802 13416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b6e172a2f5d4262a2994b1b5e478808: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:36.267025 13416 log.cc:824] T 00000000000000000000000000000000 P 4b6e172a2f5d4262a2994b1b5e478808: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:36.270514 13416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b6e172a2f5d4262a2994b1b5e478808: No bootstrap required, opened a new log
I20260924 03:21:36.272176 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:36.246118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52824 (local address 127.10.12.190:43619)
0924 03:21:36.246900 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:36.246913 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:36.246957 (+ 44us) server_negotiation.cc:407] Connection header received
0924 03:21:36.248827 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:36.248853 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:36.249312 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:36.249728 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:36.251559 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:36.253148 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:36.255164 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:36.256830 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:36.257010 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:36.262643 (+ 5633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:36.262654 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:36.262673 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:36.262727 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:36.266693 (+ 3966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:36.267901 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:36.267913 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:36.267924 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:36.268318 (+ 394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:36.268810 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:36.268814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:36.268817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:36.269412 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:36.269737 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:36.270226 (+ 489us) server_negotiation.cc:299] Negotiation successful
0924 03:21:36.271010 (+ 784us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":365,"threads_started":1}
I20260924 03:21:36.280608 13416 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b6e172a2f5d4262a2994b1b5e478808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6e172a2f5d4262a2994b1b5e478808" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 43619 } }
I20260924 03:21:36.281032 13416 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b6e172a2f5d4262a2994b1b5e478808 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:36.281132 13416 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b6e172a2f5d4262a2994b1b5e478808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b6e172a2f5d4262a2994b1b5e478808, State: Initialized, Role: FOLLOWER
I20260924 03:21:36.281785 13416 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b6e172a2f5d4262a2994b1b5e478808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6e172a2f5d4262a2994b1b5e478808" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 43619 } }
I20260924 03:21:36.282075 13416 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b6e172a2f5d4262a2994b1b5e478808 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:21:36.282229 13416 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b6e172a2f5d4262a2994b1b5e478808 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:21:36.282411 13416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b6e172a2f5d4262a2994b1b5e478808 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:36.284897 13416 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b6e172a2f5d4262a2994b1b5e478808 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6e172a2f5d4262a2994b1b5e478808" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 43619 } }
I20260924 03:21:36.285424 13416 leader_election.cc:302] T 00000000000000000000000000000000 P 4b6e172a2f5d4262a2994b1b5e478808 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b6e172a2f5d4262a2994b1b5e478808; no voters:
I20260924 03:21:36.286016 13416 leader_election.cc:288] T 00000000000000000000000000000000 P 4b6e172a2f5d4262a2994b1b5e478808 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:21:36.286170 13421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b6e172a2f5d4262a2994b1b5e478808 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:36.286864 13421 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b6e172a2f5d4262a2994b1b5e478808 [term 1 LEADER]: Becoming Leader. State: Replica: 4b6e172a2f5d4262a2994b1b5e478808, State: Running, Role: LEADER
I20260924 03:21:36.287400 13421 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b6e172a2f5d4262a2994b1b5e478808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6e172a2f5d4262a2994b1b5e478808" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 43619 } }
I20260924 03:21:36.288153 13416 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b6e172a2f5d4262a2994b1b5e478808 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:21:36.292248 13423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b6e172a2f5d4262a2994b1b5e478808 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b6e172a2f5d4262a2994b1b5e478808. Latest consensus state: current_term: 1 leader_uuid: "4b6e172a2f5d4262a2994b1b5e478808" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6e172a2f5d4262a2994b1b5e478808" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 43619 } } }
I20260924 03:21:36.292701 13423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b6e172a2f5d4262a2994b1b5e478808 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:36.292445 13422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b6e172a2f5d4262a2994b1b5e478808 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b6e172a2f5d4262a2994b1b5e478808" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6e172a2f5d4262a2994b1b5e478808" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 43619 } } }
I20260924 03:21:36.292845 13422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b6e172a2f5d4262a2994b1b5e478808 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:36.295909 13430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:21:36.302114 13430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:21:36.313441 13430 catalog_manager.cc:1347] Generated new cluster ID: d74a36c1628d41d8883b552f5ad2d821
I20260924 03:21:36.313587 13430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:21:36.327303 13430 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:21:36.329100 13430 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:21:36.343743 13430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b6e172a2f5d4262a2994b1b5e478808: Generated new TSK 0
I20260924 03:21:36.344791 13430 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
I20260924 03:21:36.409068 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:43619
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:36.785595 13442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:36.785929 13442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:36.786026 13442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:36.794278 13442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:36.794462 13442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:36.794605 13442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:21:36.806085 13442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:43619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:36.807399 13442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:36.808765 13442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:36.822310 13447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:36.822341 13448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:36.822465 13450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:36.823081 13442 server_base.cc:1034] running on GCE node
I20260924 03:21:36.823719 13442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:36.824951 13442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:36.826256 13442 hybrid_clock.cc:648] HybridClock initialized: now 1790220096826128 us; error 65 us; skew 500 ppm
I20260924 03:21:36.826586 13442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:36 dist-test-slave-00qs krb5kdc[13336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220096, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:36.829927 13442 init.cc:377] Logged in from keytab as kudu/127.10.12.129@KRBTEST.COM (short username kudu)
I20260924 03:21:36.831797 13442 webserver.cc:460] Webserver started at http://127.10.12.129:46313/ using document root <none> and password file <none>
I20260924 03:21:36.832568 13442 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:36.832681 13442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:36.832955 13442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:36.836047 13442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "b39e2b3ac28a4a358ad47d23d41fb80c"
format_stamp: "Formatted at 2026-09-24 03:21:36 on dist-test-slave-00qs"
server_key: "92d9a2f25a0d50acc950c336e573a185"
server_key_iv: "3eb13b803e1c31b39db8d11d6275df3c"
server_key_version: "encryptionkey@0"
I20260924 03:21:36.836861 13442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "b39e2b3ac28a4a358ad47d23d41fb80c"
format_stamp: "Formatted at 2026-09-24 03:21:36 on dist-test-slave-00qs"
server_key: "92d9a2f25a0d50acc950c336e573a185"
server_key_iv: "3eb13b803e1c31b39db8d11d6275df3c"
server_key_version: "encryptionkey@0"
I20260924 03:21:36.843351 13442 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260924 03:21:36.847594 13457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:36.849270 13442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:21:36.849460 13442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "b39e2b3ac28a4a358ad47d23d41fb80c"
format_stamp: "Formatted at 2026-09-24 03:21:36 on dist-test-slave-00qs"
server_key: "92d9a2f25a0d50acc950c336e573a185"
server_key_iv: "3eb13b803e1c31b39db8d11d6275df3c"
server_key_version: "encryptionkey@0"
I20260924 03:21:36.849673 13442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:36.877054 13442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:36.882114 13442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:36.882454 13442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:36.883543 13442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:21:36.885048 13442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:36.885128 13442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:36.885214 13442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:36.885267 13442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:36.921806 13442 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:36741
I20260924 03:21:36.921881 13569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:36741 every 8 connection(s)
I20260924 03:21:36.923092 13442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:21:36.923936 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 13442
I20260924 03:21:36.924160 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:21:36.924865 10290 external_mini_cluster.cc:1428] Setting key b8f388d870277a86e37ae91ccf598baf
Sep 24 03:21:36 dist-test-slave-00qs krb5kdc[13336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220096, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.130@KRBTEST.COM" created.
I20260924 03:21:36.948911 13572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:36.926415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:53055 (local address 127.10.12.190:43619)
0924 03:21:36.927049 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:36.927057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:36.928363 (+ 1306us) server_negotiation.cc:407] Connection header received
0924 03:21:36.930337 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:36.930342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:36.930452 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:36.930551 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:36.932324 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:36.933074 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:36.935238 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:36.935646 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:36.935766 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:36.939791 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:36.939797 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:36.939802 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:36.939843 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:36.943004 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:36.944383 (+ 1379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:36.944388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:36.944391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:36.944577 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:36.945227 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:36.945438 (+ 211us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:36.945443 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:36.946077 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:36.948062 (+ 1985us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:36.948290 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 03:21:36.948561 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":308,"threads_started":1}
I20260924 03:21:36.949394 13573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:36.927061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:43619 (local address 127.10.12.129:53055)
0924 03:21:36.928185 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:36.928230 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:36.930012 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:36.930852 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:36.930861 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:36.931433 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:36.932111 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:36.932133 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:36.933243 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:36.933247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:36.935071 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:36.935087 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:36.935905 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:36.935912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:36.936220 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:36.936910 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:36.936935 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:36.939472 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:36.943252 (+ 3780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:36.943265 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:36.943290 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:36.944168 (+ 878us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:36.944774 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:36.944780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:36.944784 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:36.945065 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:36.946379 (+ 1314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:36.946390 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:36.946940 (+ 550us) client_negotiation.cc:769] Sending connection context
0924 03:21:36.947333 (+ 393us) client_negotiation.cc:240] Negotiation successful
0924 03:21:36.947855 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":475,"threads_started":1}
I20260924 03:21:36.954036 13570 heartbeater.cc:347] Connected to a master server at 127.10.12.190:43619
I20260924 03:21:36.954499 13570 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:36.955440 13570 heartbeater.cc:511] Master 127.10.12.190:43619 requested a full tablet report, sending...
I20260924 03:21:36.958117 13381 ts_manager.cc:194] Registered new tserver with Master: b39e2b3ac28a4a358ad47d23d41fb80c (127.10.12.129:36741)
I20260924 03:21:36.959724 13381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.129@KRBTEST.COM'} at 127.10.12.129:53055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
I20260924 03:21:36.976737 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:43619
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:37.331264 13576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:37.331565 13576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:37.331661 13576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:37.339555 13576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:37.339697 13576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:37.339813 13576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:21:37.350598 13576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:43619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:37.352017 13576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:37.353405 13576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:37.368281 13581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:37.368114 13584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:37.368229 13582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:37.369925 13576 server_base.cc:1034] running on GCE node
I20260924 03:21:37.370759 13576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:37.372097 13576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:37.373363 13576 hybrid_clock.cc:648] HybridClock initialized: now 1790220097373287 us; error 67 us; skew 500 ppm
I20260924 03:21:37.373797 13576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:37 dist-test-slave-00qs krb5kdc[13336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220097, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:37.377560 13576 init.cc:377] Logged in from keytab as kudu/127.10.12.130@KRBTEST.COM (short username kudu)
I20260924 03:21:37.379544 13576 webserver.cc:460] Webserver started at http://127.10.12.130:41963/ using document root <none> and password file <none>
I20260924 03:21:37.380412 13576 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:37.380554 13576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:37.380860 13576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:37.384109 13576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "efe82cd78f3a486ebf05ec37a372048d"
format_stamp: "Formatted at 2026-09-24 03:21:37 on dist-test-slave-00qs"
server_key: "6e43042d3a477ea93ccbea23cb5cb893"
server_key_iv: "5a0e2295ccc848fde981869007d16527"
server_key_version: "encryptionkey@0"
I20260924 03:21:37.385267 13576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "efe82cd78f3a486ebf05ec37a372048d"
format_stamp: "Formatted at 2026-09-24 03:21:37 on dist-test-slave-00qs"
server_key: "6e43042d3a477ea93ccbea23cb5cb893"
server_key_iv: "5a0e2295ccc848fde981869007d16527"
server_key_version: "encryptionkey@0"
I20260924 03:21:37.392118 13576 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 03:21:37.396517 13591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:37.398324 13576 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 03:21:37.398524 13576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "efe82cd78f3a486ebf05ec37a372048d"
format_stamp: "Formatted at 2026-09-24 03:21:37 on dist-test-slave-00qs"
server_key: "6e43042d3a477ea93ccbea23cb5cb893"
server_key_iv: "5a0e2295ccc848fde981869007d16527"
server_key_version: "encryptionkey@0"
I20260924 03:21:37.398756 13576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:37.418689 13576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:37.424283 13576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:37.424615 13576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:37.425750 13576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:21:37.427412 13576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:37.427515 13576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:37.427618 13576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:37.427672 13576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:37.465171 13576 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:45263
I20260924 03:21:37.465206 13703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:45263 every 8 connection(s)
I20260924 03:21:37.466997 13576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:21:37.468173 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 13576
I20260924 03:21:37.468408 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:21:37.468956 10290 external_mini_cluster.cc:1428] Setting key 44692e07106d548316e1c009e17692b9
Sep 24 03:21:37 dist-test-slave-00qs krb5kdc[13336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220097, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.131@KRBTEST.COM" created.
I20260924 03:21:37.498492 13706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:37.470790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:55863 (local address 127.10.12.190:43619)
0924 03:21:37.471328 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:37.471334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:37.474993 (+ 3659us) server_negotiation.cc:407] Connection header received
0924 03:21:37.477898 (+ 2905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:37.477904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:37.477997 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:37.478100 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:37.480094 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:37.481333 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:37.484108 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:37.484707 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:37.484891 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:37.490185 (+ 5294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:37.490191 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:37.490195 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:37.490240 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:37.492877 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:37.494655 (+ 1778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:37.494663 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:37.494665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:37.494899 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:37.495764 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:37.495770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:37.495773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:37.496520 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:37.496739 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:37.497829 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0924 03:21:37.498138 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":303,"threads_started":1}
I20260924 03:21:37.499577 13707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:37.471421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:43619 (local address 127.10.12.130:55863)
0924 03:21:37.474757 (+ 3336us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:37.474816 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:37.477436 (+ 2620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:37.478395 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:37.478406 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:37.479019 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:37.479820 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:37.479847 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:37.481712 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:37.481718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:37.483696 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:37.483719 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:37.485049 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:37.485056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:37.485398 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:37.486326 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:37.486365 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:37.489870 (+ 3505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:37.493079 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:37.493217 (+ 138us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:37.493249 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:37.494323 (+ 1074us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:37.495070 (+ 747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:37.495075 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:37.495079 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:37.495552 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:37.496713 (+ 1161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:37.496723 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:37.497260 (+ 537us) client_negotiation.cc:769] Sending connection context
0924 03:21:37.497624 (+ 364us) client_negotiation.cc:240] Negotiation successful
0924 03:21:37.498248 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2981,"thread_start_us":2743,"threads_started":1}
I20260924 03:21:37.501250 13704 heartbeater.cc:347] Connected to a master server at 127.10.12.190:43619
I20260924 03:21:37.501564 13704 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:37.502347 13704 heartbeater.cc:511] Master 127.10.12.190:43619 requested a full tablet report, sending...
I20260924 03:21:37.503909 13381 ts_manager.cc:194] Registered new tserver with Master: efe82cd78f3a486ebf05ec37a372048d (127.10.12.130:45263)
I20260924 03:21:37.504843 13381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.130@KRBTEST.COM'} at 127.10.12.130:55863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
I20260924 03:21:37.521790 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:43619
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:37.861780 13710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:37.862099 13710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:37.862192 13710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:37.869907 13710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:37.870034 13710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:37.870151 13710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:21:37.881922 13710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:43619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:37.883287 13710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:37.884719 13710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:37.900620 13718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:37.900713 13715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:37.900704 13716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:37.901585 13710 server_base.cc:1034] running on GCE node
I20260924 03:21:37.902285 13710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:37.903553 13710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:37.904942 13710 hybrid_clock.cc:648] HybridClock initialized: now 1790220097904844 us; error 97 us; skew 500 ppm
I20260924 03:21:37.905509 13710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:37 dist-test-slave-00qs krb5kdc[13336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220097, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:37.910005 13710 init.cc:377] Logged in from keytab as kudu/127.10.12.131@KRBTEST.COM (short username kudu)
I20260924 03:21:37.911850 13710 webserver.cc:460] Webserver started at http://127.10.12.131:45273/ using document root <none> and password file <none>
I20260924 03:21:37.912662 13710 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:37.912801 13710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:37.913111 13710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:37.916364 13710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "5a57a9cd8122421d8208f106326cbbbb"
format_stamp: "Formatted at 2026-09-24 03:21:37 on dist-test-slave-00qs"
server_key: "49ff5a926beabca1234f4719b750db01"
server_key_iv: "a54cc31ec2e0722aebd826f261e1cac1"
server_key_version: "encryptionkey@0"
I20260924 03:21:37.917207 13710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "5a57a9cd8122421d8208f106326cbbbb"
format_stamp: "Formatted at 2026-09-24 03:21:37 on dist-test-slave-00qs"
server_key: "49ff5a926beabca1234f4719b750db01"
server_key_iv: "a54cc31ec2e0722aebd826f261e1cac1"
server_key_version: "encryptionkey@0"
I20260924 03:21:37.923309 13710 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 03:21:37.927623 13725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:37.929422 13710 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:21:37.929643 13710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "5a57a9cd8122421d8208f106326cbbbb"
format_stamp: "Formatted at 2026-09-24 03:21:37 on dist-test-slave-00qs"
server_key: "49ff5a926beabca1234f4719b750db01"
server_key_iv: "a54cc31ec2e0722aebd826f261e1cac1"
server_key_version: "encryptionkey@0"
I20260924 03:21:37.929872 13710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:37.955936 13710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:37.960830 13710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:37.961140 13710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:37.962255 13710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:21:37.963419 13570 heartbeater.cc:503] Master 127.10.12.190:43619 was elected leader, sending a full tablet report...
I20260924 03:21:37.963769 13710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:37.963857 13710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:37.963987 13710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:37.964042 13710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260924 03:21:38.012441 13710 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:45689
I20260924 03:21:38.012485 13837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:45689 every 8 connection(s)
I20260924 03:21:38.014251 13710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:21:38.015308 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 13710
I20260924 03:21:38.015501 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:21:38.016047 10290 external_mini_cluster.cc:1428] Setting key 63d570b841c0968b09656d339d7af12b
Sep 24 03:21:38 dist-test-slave-00qs krb5kdc[13336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220097, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:38.039073 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:38.017518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:47495 (local address 127.10.12.190:43619)
0924 03:21:38.017980 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:38.017987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:38.020033 (+ 2046us) server_negotiation.cc:407] Connection header received
0924 03:21:38.022757 (+ 2724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:38.022779 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:38.022919 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:38.023231 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:38.025249 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:38.026156 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:38.027922 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:38.028409 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:38.028526 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:38.032585 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:38.032591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:38.032594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:38.032631 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:38.035761 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:38.036638 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:38.036642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:38.036645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:38.036776 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:38.037233 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:38.037236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:38.037238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:38.037533 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:38.037683 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:38.038426 (+ 743us) server_negotiation.cc:299] Negotiation successful
0924 03:21:38.038669 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":220,"threads_started":1}
I20260924 03:21:38.040091 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:38.018268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:43619 (local address 127.10.12.131:47495)
0924 03:21:38.019744 (+ 1476us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:38.019804 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:38.022337 (+ 2533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:38.023489 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:38.023500 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:38.024323 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:38.024993 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:38.025015 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:38.026351 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:38.026357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:38.027709 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:38.027725 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:38.028680 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:38.028685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:38.028982 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:38.029693 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:38.029717 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:38.032352 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:38.035929 (+ 3577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:38.035940 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:38.036003 (+ 63us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:38.036493 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:38.036888 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:38.036894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:38.036898 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:38.037111 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:38.037668 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:38.037677 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:38.038050 (+ 373us) client_negotiation.cc:769] Sending connection context
0924 03:21:38.038305 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 03:21:38.038867 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"thread_start_us":494,"threads_started":1}
I20260924 03:21:38.041638 13838 heartbeater.cc:347] Connected to a master server at 127.10.12.190:43619
I20260924 03:21:38.041977 13838 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:38.042725 13838 heartbeater.cc:511] Master 127.10.12.190:43619 requested a full tablet report, sending...
I20260924 03:21:38.044678 13381 ts_manager.cc:194] Registered new tserver with Master: 5a57a9cd8122421d8208f106326cbbbb (127.10.12.131:45689)
I20260924 03:21:38.045835 13381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.131@KRBTEST.COM'} at 127.10.12.131:47495
I20260924 03:21:38.046936 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 03:21:38 dist-test-slave-00qs krb5kdc[13336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220098, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 03:21:38 dist-test-slave-00qs krb5kdc[13336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220098, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:38.102438 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:38.081387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52836 (local address 127.10.12.190:43619)
0924 03:21:38.081556 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:38.081562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:38.082039 (+ 477us) server_negotiation.cc:407] Connection header received
0924 03:21:38.082228 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:38.082233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:38.082437 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:38.082549 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:38.084220 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:38.085027 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:38.086836 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:38.087275 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:38.087399 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:38.093211 (+ 5812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:38.093219 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:38.093224 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:38.093279 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:38.097853 (+ 4574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:38.098939 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:38.098944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:38.098946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:38.099115 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:38.099871 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:38.099878 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:38.099881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:38.100704 (+ 823us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:38.101232 (+ 528us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:38.101583 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 03:21:38.101889 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 03:21:38.103349 13381 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52836
I20260924 03:21:38.105968 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 13442
Sep 24 03:21:38 dist-test-slave-00qs krb5kdc[13336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220098, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.12.129@KRBTEST.COM
I20260924 03:21:38.125180 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:38.107271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44116 (local address 127.10.12.129:36741)
0924 03:21:38.108083 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:38.108092 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:38.108120 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:21:38.108196 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:38.108202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:38.108434 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:38.108591 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:38.109952 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:38.111032 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:38.112412 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:38.113062 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:38.113189 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:38.118194 (+ 5005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:38.118213 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:38.118230 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:38.118277 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:38.121575 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:38.122463 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:38.122472 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:38.122474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:38.122617 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:38.123025 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:38.123028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:38.123030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:38.123387 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:38.123629 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:38.124340 (+ 711us) server_negotiation.cc:299] Negotiation successful
0924 03:21:38.124766 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":419,"threads_started":1}
W20260924 03:21:38.126763 13565 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44116
W20260924 03:21:38.128028 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 13442: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:21:38.128257 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 13442
I20260924 03:21:38.141463 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 13576
Sep 24 03:21:38 dist-test-slave-00qs krb5kdc[13336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220098, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.12.130@KRBTEST.COM
I20260924 03:21:38.160298 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:38.142751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40348 (local address 127.10.12.130:45263)
0924 03:21:38.143445 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:38.143453 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:38.143487 (+ 34us) server_negotiation.cc:407] Connection header received
0924 03:21:38.143555 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:38.143562 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:38.143851 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:38.144110 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:38.145528 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:38.146544 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:38.147888 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:38.148537 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:38.148668 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:38.153460 (+ 4792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:38.153479 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:38.153496 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:38.153543 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:38.156718 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:38.157611 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:38.157619 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:38.157622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:38.157785 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:38.158237 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:38.158241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:38.158243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:38.158633 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:38.158981 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:38.159606 (+ 625us) server_negotiation.cc:299] Negotiation successful
0924 03:21:38.159860 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":321,"threads_started":1}
W20260924 03:21:38.161891 13699 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40348
W20260924 03:21:38.163836 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 13576: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:21:38.164242 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 13576
I20260924 03:21:38.176774 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 13710
Sep 24 03:21:38 dist-test-slave-00qs krb5kdc[13336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220098, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.12.131@KRBTEST.COM
I20260924 03:21:38.197201 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:38.177887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50468 (local address 127.10.12.131:45689)
0924 03:21:38.178538 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:38.178547 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:38.178586 (+ 39us) server_negotiation.cc:407] Connection header received
0924 03:21:38.178652 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:38.178659 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:38.178972 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:38.179122 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:38.180811 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:38.181963 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:38.183362 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:38.184022 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:38.184163 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:38.189061 (+ 4898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:38.189080 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:38.189099 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:38.189156 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:38.192470 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:38.193680 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:38.193691 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:38.193695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:38.193867 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:38.194659 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:38.194663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:38.194665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:38.195098 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:38.195454 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:38.196408 (+ 954us) server_negotiation.cc:299] Negotiation successful
0924 03:21:38.196831 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":310,"threads_started":1}
W20260924 03:21:38.198740 13833 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50468
W20260924 03:21:38.200374 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 13710: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:21:38.200578 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 13710
I20260924 03:21:38.213383 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 13349
I20260924 03:21:38.214881 13411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:38.320369 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 13349
2026-09-24T03:21:38Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3768 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:21:38 dist-test-slave-00qs krb5kdc[13856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:21:38 dist-test-slave-00qs krb5kdc[13856](info): set up 2 sockets
Sep 24 03:21:38 dist-test-slave-00qs krb5kdc[13856](info): commencing operation
krb5kdc: starting...
W20260924 03:21:39.340442 10290 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:21:39 dist-test-slave-00qs krb5kdc[13856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:21:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:21:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:21:39.548647 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:44829
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:44829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:39.880937 13869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:39.881285 13869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:39.881376 13869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:39.889365 13869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:21:39.889513 13869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:39.889561 13869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:39.889616 13869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:21:39.889660 13869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:21:39.900352 13869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:44829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:44829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:39.901680 13869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:39.903008 13869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:39.915086 13874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:39.915222 13875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:39.915445 13877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:39.916023 13869 server_base.cc:1034] running on GCE node
I20260924 03:21:39.916817 13869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:39.918325 13869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:39.919567 13869 hybrid_clock.cc:648] HybridClock initialized: now 1790220099919503 us; error 54 us; skew 500 ppm
I20260924 03:21:39.919986 13869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:39 dist-test-slave-00qs krb5kdc[13856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220099, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:39.928987 13869 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:21:39.930665 13869 webserver.cc:460] Webserver started at http://127.10.12.190:40353/ using document root <none> and password file <none>
I20260924 03:21:39.931414 13869 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:39.931525 13869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:39.931828 13869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:39.934984 13869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "23c15e120cfd40c7b529ab5bbf0e418c"
format_stamp: "Formatted at 2026-09-24 03:21:39 on dist-test-slave-00qs"
server_key: "bb4fbb61893abdd20e38aaf92bde3d83"
server_key_iv: "1a038a032d7d51f75aa96941945cd1a4"
server_key_version: "encryptionkey@0"
I20260924 03:21:39.935812 13869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "23c15e120cfd40c7b529ab5bbf0e418c"
format_stamp: "Formatted at 2026-09-24 03:21:39 on dist-test-slave-00qs"
server_key: "bb4fbb61893abdd20e38aaf92bde3d83"
server_key_iv: "1a038a032d7d51f75aa96941945cd1a4"
server_key_version: "encryptionkey@0"
I20260924 03:21:39.941393 13869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 03:21:39.945017 13884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:39.946489 13869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:21:39.946660 13869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "23c15e120cfd40c7b529ab5bbf0e418c"
format_stamp: "Formatted at 2026-09-24 03:21:39 on dist-test-slave-00qs"
server_key: "bb4fbb61893abdd20e38aaf92bde3d83"
server_key_iv: "1a038a032d7d51f75aa96941945cd1a4"
server_key_version: "encryptionkey@0"
I20260924 03:21:39.946933 13869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:39.986178 13869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:39.991508 13869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:39.991906 13869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:40.013609 13869 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:44829
I20260924 03:21:40.013626 13935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:44829 every 8 connection(s)
I20260924 03:21:40.015228 13869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:21:40.019522 13936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:40.019590 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 13869
I20260924 03:21:40.019789 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:21:40.020460 10290 external_mini_cluster.cc:1428] Setting key 9165914ba31097f8241280d301f417a9
I20260924 03:21:40.034952 13936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23c15e120cfd40c7b529ab5bbf0e418c: Bootstrap starting.
Sep 24 03:21:40 dist-test-slave-00qs krb5kdc[13856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:40.038506 13936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23c15e120cfd40c7b529ab5bbf0e418c: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:40.039883 13936 log.cc:824] T 00000000000000000000000000000000 P 23c15e120cfd40c7b529ab5bbf0e418c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:40.043829 13936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23c15e120cfd40c7b529ab5bbf0e418c: No bootstrap required, opened a new log
I20260924 03:21:40.048815 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:40.022802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53764 (local address 127.10.12.190:44829)
0924 03:21:40.023662 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:40.023683 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:40.023757 (+ 74us) server_negotiation.cc:407] Connection header received
0924 03:21:40.026373 (+ 2616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:40.026412 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:40.027086 (+ 674us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:40.027618 (+ 532us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:40.029254 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:40.031387 (+ 2133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:40.032919 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:40.034038 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:40.034233 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:40.039035 (+ 4802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:40.039047 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:40.039069 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:40.039127 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:40.043486 (+ 4359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:40.044458 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:40.044468 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:40.044478 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:40.044718 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:40.045185 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:40.045191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:40.045193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:40.045959 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:40.046271 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:40.046743 (+ 472us) server_negotiation.cc:299] Negotiation successful
0924 03:21:40.047264 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":324,"threads_started":1}
I20260924 03:21:40.054540 13936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23c15e120cfd40c7b529ab5bbf0e418c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c15e120cfd40c7b529ab5bbf0e418c" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44829 } }
I20260924 03:21:40.054951 13936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23c15e120cfd40c7b529ab5bbf0e418c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:40.055032 13936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23c15e120cfd40c7b529ab5bbf0e418c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c15e120cfd40c7b529ab5bbf0e418c, State: Initialized, Role: FOLLOWER
I20260924 03:21:40.055545 13936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23c15e120cfd40c7b529ab5bbf0e418c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c15e120cfd40c7b529ab5bbf0e418c" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44829 } }
I20260924 03:21:40.055763 13936 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23c15e120cfd40c7b529ab5bbf0e418c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:21:40.055861 13936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23c15e120cfd40c7b529ab5bbf0e418c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:21:40.056035 13936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23c15e120cfd40c7b529ab5bbf0e418c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:40.058248 13936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23c15e120cfd40c7b529ab5bbf0e418c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c15e120cfd40c7b529ab5bbf0e418c" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44829 } }
I20260924 03:21:40.058880 13936 leader_election.cc:302] T 00000000000000000000000000000000 P 23c15e120cfd40c7b529ab5bbf0e418c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23c15e120cfd40c7b529ab5bbf0e418c; no voters:
I20260924 03:21:40.059533 13936 leader_election.cc:288] T 00000000000000000000000000000000 P 23c15e120cfd40c7b529ab5bbf0e418c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:21:40.059676 13941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23c15e120cfd40c7b529ab5bbf0e418c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:40.060534 13941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23c15e120cfd40c7b529ab5bbf0e418c [term 1 LEADER]: Becoming Leader. State: Replica: 23c15e120cfd40c7b529ab5bbf0e418c, State: Running, Role: LEADER
I20260924 03:21:40.061082 13941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23c15e120cfd40c7b529ab5bbf0e418c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c15e120cfd40c7b529ab5bbf0e418c" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44829 } }
I20260924 03:21:40.061815 13936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23c15e120cfd40c7b529ab5bbf0e418c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:21:40.064368 13943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23c15e120cfd40c7b529ab5bbf0e418c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23c15e120cfd40c7b529ab5bbf0e418c. Latest consensus state: current_term: 1 leader_uuid: "23c15e120cfd40c7b529ab5bbf0e418c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c15e120cfd40c7b529ab5bbf0e418c" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44829 } } }
I20260924 03:21:40.064868 13943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23c15e120cfd40c7b529ab5bbf0e418c [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:40.068082 13948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:21:40.068495 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23c15e120cfd40c7b529ab5bbf0e418c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23c15e120cfd40c7b529ab5bbf0e418c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c15e120cfd40c7b529ab5bbf0e418c" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44829 } } }
I20260924 03:21:40.068902 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23c15e120cfd40c7b529ab5bbf0e418c [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:40.073680 13948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:21:40.082860 13948 catalog_manager.cc:1347] Generated new cluster ID: 545f919b752c4d239cd9bff345a046bd
I20260924 03:21:40.082998 13948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:21:40.100642 13948 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:21:40.102643 13948 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:21:40.113670 13948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23c15e120cfd40c7b529ab5bbf0e418c: Generated new TSK 0
I20260924 03:21:40.114571 13948 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
I20260924 03:21:40.173326 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:44829
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:40.535573 13962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:40.535907 13962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:40.536082 13962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:40.543710 13962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:40.543870 13962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:40.544066 13962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:21:40.554797 13962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:44829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:40.556289 13962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:40.557734 13962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:40.571781 13968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:40.571785 13967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:40.572381 13970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:40.578231 13962 server_base.cc:1034] running on GCE node
I20260924 03:21:40.578969 13962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:40.580013 13962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:40.581250 13962 hybrid_clock.cc:648] HybridClock initialized: now 1790220100581169 us; error 74 us; skew 500 ppm
I20260924 03:21:40.581612 13962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:40 dist-test-slave-00qs krb5kdc[13856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220100, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:40.585089 13962 init.cc:377] Logged in from keytab as kudu/127.10.12.129@KRBTEST.COM (short username kudu)
I20260924 03:21:40.586865 13962 webserver.cc:460] Webserver started at http://127.10.12.129:40613/ using document root <none> and password file <none>
I20260924 03:21:40.587651 13962 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:40.587780 13962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:40.588261 13962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:40.592125 13962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "6269e0bf019f49a1b54d07275d7b0ff4"
format_stamp: "Formatted at 2026-09-24 03:21:40 on dist-test-slave-00qs"
server_key: "9694573cfa2079912dd4fee3a94f0185"
server_key_iv: "bbb6ff779f747ae69bfc23c0a88d8562"
server_key_version: "encryptionkey@0"
I20260924 03:21:40.593140 13962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "6269e0bf019f49a1b54d07275d7b0ff4"
format_stamp: "Formatted at 2026-09-24 03:21:40 on dist-test-slave-00qs"
server_key: "9694573cfa2079912dd4fee3a94f0185"
server_key_iv: "bbb6ff779f747ae69bfc23c0a88d8562"
server_key_version: "encryptionkey@0"
I20260924 03:21:40.600059 13962 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260924 03:21:40.604028 13977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:40.605777 13962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:21:40.605978 13962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "6269e0bf019f49a1b54d07275d7b0ff4"
format_stamp: "Formatted at 2026-09-24 03:21:40 on dist-test-slave-00qs"
server_key: "9694573cfa2079912dd4fee3a94f0185"
server_key_iv: "bbb6ff779f747ae69bfc23c0a88d8562"
server_key_version: "encryptionkey@0"
I20260924 03:21:40.606195 13962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:40.623526 13962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:40.627980 13962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:40.628273 13962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:40.629328 13962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:21:40.630777 13962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:40.630860 13962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:40.630980 13962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:40.631036 13962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:40.666055 13962 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:45157
I20260924 03:21:40.666088 14089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:45157 every 8 connection(s)
I20260924 03:21:40.667479 13962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:21:40.668919 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 13962
I20260924 03:21:40.669148 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:21:40.669946 10290 external_mini_cluster.cc:1428] Setting key bcbe7d16d00a53bb07fed4c983652baf
Sep 24 03:21:40 dist-test-slave-00qs krb5kdc[13856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220100, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.12.130@KRBTEST.COM; defaulting to no policy
I20260924 03:21:40.691924 14092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:40.670958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:40509 (local address 127.10.12.190:44829)
0924 03:21:40.671921 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:40.671928 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:40.672702 (+ 774us) server_negotiation.cc:407] Connection header received
0924 03:21:40.674566 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:40.674572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:40.674651 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:40.674763 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:40.676422 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:40.677289 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:40.679669 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:40.680187 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:40.680314 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:40.684668 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:40.684677 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:40.684683 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:40.684732 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:40.688065 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:40.689171 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:40.689178 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:40.689182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:40.689446 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:40.690078 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:40.690082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:40.690084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:40.690401 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:40.690532 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:40.691475 (+ 943us) server_negotiation.cc:299] Negotiation successful
0924 03:21:40.691673 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843,"thread_start_us":729,"threads_started":1}
I20260924 03:21:40.693038 14093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:40.671648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:44829 (local address 127.10.12.129:40509)
0924 03:21:40.672525 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:40.672579 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:40.674291 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:40.674963 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:40.674972 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:40.675583 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:40.676249 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:40.676271 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:40.677466 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:40.677473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:40.679481 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:40.679498 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:40.680432 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:40.680438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:40.680728 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:40.681498 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:40.681528 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:40.684293 (+ 2765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:40.688289 (+ 3996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:40.688323 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:40.688340 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:40.688844 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:40.689614 (+ 770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:40.689620 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:40.689623 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:40.689917 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:40.690567 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:40.690575 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:40.691029 (+ 454us) client_negotiation.cc:769] Sending connection context
0924 03:21:40.691297 (+ 268us) client_negotiation.cc:240] Negotiation successful
0924 03:21:40.691796 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":333,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.130@KRBTEST.COM" created.
I20260924 03:21:40.695140 14090 heartbeater.cc:347] Connected to a master server at 127.10.12.190:44829
I20260924 03:21:40.695511 14090 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:40.696487 14090 heartbeater.cc:511] Master 127.10.12.190:44829 requested a full tablet report, sending...
I20260924 03:21:40.699410 13901 ts_manager.cc:194] Registered new tserver with Master: 6269e0bf019f49a1b54d07275d7b0ff4 (127.10.12.129:45157)
I20260924 03:21:40.701028 13901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.129@KRBTEST.COM'} at 127.10.12.129:40509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
I20260924 03:21:40.725339 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:44829
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:41.068353 14096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:41.068648 14096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:41.068742 14096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:41.076356 14096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:41.076475 14096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:41.076602 14096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:21:41.086783 14096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:44829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:41.088119 14096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:41.089375 14096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:41.102952 14101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:41.102969 14102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:41.102957 14104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:41.103825 14096 server_base.cc:1034] running on GCE node
I20260924 03:21:41.104493 14096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:41.105492 14096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:41.106794 14096 hybrid_clock.cc:648] HybridClock initialized: now 1790220101106667 us; error 51 us; skew 500 ppm
I20260924 03:21:41.107118 14096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:41 dist-test-slave-00qs krb5kdc[13856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220101, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:41.110707 14096 init.cc:377] Logged in from keytab as kudu/127.10.12.130@KRBTEST.COM (short username kudu)
I20260924 03:21:41.112389 14096 webserver.cc:460] Webserver started at http://127.10.12.130:42257/ using document root <none> and password file <none>
I20260924 03:21:41.113085 14096 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:41.113189 14096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:41.113463 14096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:41.116786 14096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "51952fb366ed42189b483addea64f08f"
format_stamp: "Formatted at 2026-09-24 03:21:41 on dist-test-slave-00qs"
server_key: "8317d43219fb64e59e04d9e9ec18b17d"
server_key_iv: "60ccdec0acea5a979e4534986fd2a3a2"
server_key_version: "encryptionkey@0"
I20260924 03:21:41.117600 14096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "51952fb366ed42189b483addea64f08f"
format_stamp: "Formatted at 2026-09-24 03:21:41 on dist-test-slave-00qs"
server_key: "8317d43219fb64e59e04d9e9ec18b17d"
server_key_iv: "60ccdec0acea5a979e4534986fd2a3a2"
server_key_version: "encryptionkey@0"
I20260924 03:21:41.123298 14096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:21:41.127168 14111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:41.128705 14096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:21:41.128861 14096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "51952fb366ed42189b483addea64f08f"
format_stamp: "Formatted at 2026-09-24 03:21:41 on dist-test-slave-00qs"
server_key: "8317d43219fb64e59e04d9e9ec18b17d"
server_key_iv: "60ccdec0acea5a979e4534986fd2a3a2"
server_key_version: "encryptionkey@0"
I20260924 03:21:41.129055 14096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:41.154659 14096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:41.159277 14096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:41.159539 14096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:41.160552 14096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:21:41.162066 14096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:41.162145 14096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:41.162231 14096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:41.162298 14096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:41.194593 14096 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:42753
I20260924 03:21:41.194674 14223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:42753 every 8 connection(s)
I20260924 03:21:41.195928 14096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:21:41.198037 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 14096
I20260924 03:21:41.198311 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:21:41.199142 10290 external_mini_cluster.cc:1428] Setting key a93dfe1833d14ecfb42ef3c3c6329b57
Sep 24 03:21:41 dist-test-slave-00qs krb5kdc[13856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220101, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.12.131@KRBTEST.COM; defaulting to no policy
I20260924 03:21:41.218181 14226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:41.198734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:59313 (local address 127.10.12.190:44829)
0924 03:21:41.199296 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:41.199303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:41.200323 (+ 1020us) server_negotiation.cc:407] Connection header received
0924 03:21:41.202220 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:41.202227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:41.202324 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:41.202437 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:41.204064 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:41.205027 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:41.207154 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:41.207646 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:41.207768 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:41.212146 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:41.212152 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:41.212156 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:41.212194 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:41.214906 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:41.215759 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:41.215764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:41.215767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:41.215901 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:41.216392 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:41.216396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:41.216398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:41.216666 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:41.216799 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:41.217578 (+ 779us) server_negotiation.cc:299] Negotiation successful
0924 03:21:41.217813 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":284,"threads_started":1}
I20260924 03:21:41.219170 14227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:41.199303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:44829 (local address 127.10.12.130:59313)
0924 03:21:41.200152 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:41.200205 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:41.201933 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:41.202638 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:41.202647 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:41.203269 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:41.203874 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:41.203894 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:41.205218 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:41.205222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:41.206954 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:41.206973 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:41.207912 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:41.207920 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:41.208243 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:41.208988 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:41.209018 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:41.211816 (+ 2798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:41.215093 (+ 3277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:41.215105 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:41.215123 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:41.215619 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:41.216046 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:41.216051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:41.216055 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:41.216270 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:41.216799 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:41.216819 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:41.217199 (+ 380us) client_negotiation.cc:769] Sending connection context
0924 03:21:41.217445 (+ 246us) client_negotiation.cc:240] Negotiation successful
0924 03:21:41.218022 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":313,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.131@KRBTEST.COM" created.
I20260924 03:21:41.220824 14224 heartbeater.cc:347] Connected to a master server at 127.10.12.190:44829
I20260924 03:21:41.221140 14224 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:41.221884 14224 heartbeater.cc:511] Master 127.10.12.190:44829 requested a full tablet report, sending...
I20260924 03:21:41.223505 13901 ts_manager.cc:194] Registered new tserver with Master: 51952fb366ed42189b483addea64f08f (127.10.12.130:42753)
I20260924 03:21:41.224573 13901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.130@KRBTEST.COM'} at 127.10.12.130:59313
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
I20260924 03:21:41.244796 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:44829
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:41.575314 14230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:41.575627 14230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:41.575744 14230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:41.583325 14230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:41.583455 14230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:41.583587 14230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:21:41.594424 14230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:44829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:41.595849 14230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:41.597219 14230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:41.612862 14235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:41.613371 14236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:41.613587 14238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:41.615201 14230 server_base.cc:1034] running on GCE node
I20260924 03:21:41.616055 14230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:41.617362 14230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:41.618590 14230 hybrid_clock.cc:648] HybridClock initialized: now 1790220101618516 us; error 64 us; skew 500 ppm
I20260924 03:21:41.618964 14230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:41 dist-test-slave-00qs krb5kdc[13856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220101, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:41.622888 14230 init.cc:377] Logged in from keytab as kudu/127.10.12.131@KRBTEST.COM (short username kudu)
I20260924 03:21:41.624886 14230 webserver.cc:460] Webserver started at http://127.10.12.131:40955/ using document root <none> and password file <none>
I20260924 03:21:41.625900 14230 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:41.626055 14230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:41.626459 14230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:41.630813 14230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "09fbe424f4b44d468366fdacec5097fa"
format_stamp: "Formatted at 2026-09-24 03:21:41 on dist-test-slave-00qs"
server_key: "68be3faf21afcb6ae70310d2c0d0233a"
server_key_iv: "b7252d47419760beaed6e97e44f7fbfb"
server_key_version: "encryptionkey@0"
I20260924 03:21:41.632063 14230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "09fbe424f4b44d468366fdacec5097fa"
format_stamp: "Formatted at 2026-09-24 03:21:41 on dist-test-slave-00qs"
server_key: "68be3faf21afcb6ae70310d2c0d0233a"
server_key_iv: "b7252d47419760beaed6e97e44f7fbfb"
server_key_version: "encryptionkey@0"
I20260924 03:21:41.638994 14230 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260924 03:21:41.644163 14245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:41.646080 14230 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260924 03:21:41.646301 14230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "09fbe424f4b44d468366fdacec5097fa"
format_stamp: "Formatted at 2026-09-24 03:21:41 on dist-test-slave-00qs"
server_key: "68be3faf21afcb6ae70310d2c0d0233a"
server_key_iv: "b7252d47419760beaed6e97e44f7fbfb"
server_key_version: "encryptionkey@0"
I20260924 03:21:41.646574 14230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:41.670646 14230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:41.675228 14230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:41.675498 14230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:41.676481 14230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:21:41.677865 14230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:41.677940 14230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:41.678026 14230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:41.678071 14230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:41.705060 14090 heartbeater.cc:503] Master 127.10.12.190:44829 was elected leader, sending a full tablet report...
I20260924 03:21:41.710709 14230 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:35639
I20260924 03:21:41.710767 14357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:35639 every 8 connection(s)
I20260924 03:21:41.712157 14230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:21:41.716025 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 14230
I20260924 03:21:41.716228 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:21:41.716696 10290 external_mini_cluster.cc:1428] Setting key 429415850b85e140cd293af8eafa0910
Sep 24 03:21:41 dist-test-slave-00qs krb5kdc[13856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220101, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:41.733215 14226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:41.714706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:45019 (local address 127.10.12.190:44829)
0924 03:21:41.714897 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:41.714904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:41.716164 (+ 1260us) server_negotiation.cc:407] Connection header received
0924 03:21:41.718261 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:41.718267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:41.718375 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:41.718460 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:41.720493 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:41.721231 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:41.722837 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:41.723234 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:41.723355 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:41.726985 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:41.726990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:41.726994 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:41.727030 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:41.729789 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:41.730647 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:41.730652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:41.730655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:41.730813 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:41.731283 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:41.731287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:41.731289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:41.731565 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:41.731742 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:41.732588 (+ 846us) server_negotiation.cc:299] Negotiation successful
0924 03:21:41.732838 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 03:21:41.734184 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:41.715226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:44829 (local address 127.10.12.131:45019)
0924 03:21:41.716000 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:41.716048 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:41.717974 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:41.718719 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:41.718728 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:41.719543 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:41.720319 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:41.720340 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:41.721396 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:41.721402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:41.722681 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:41.722695 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:41.723467 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:41.723474 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:41.723713 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:41.724493 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:41.724522 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:41.726721 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:41.729992 (+ 3271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:41.730000 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:41.730017 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:41.730502 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:41.730934 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:41.730939 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:41.730942 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:41.731154 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:41.731700 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:41.731709 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:41.732172 (+ 463us) client_negotiation.cc:769] Sending connection context
0924 03:21:41.732460 (+ 288us) client_negotiation.cc:240] Negotiation successful
0924 03:21:41.733055 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":281,"threads_started":1}
I20260924 03:21:41.735523 14358 heartbeater.cc:347] Connected to a master server at 127.10.12.190:44829
I20260924 03:21:41.735826 14358 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:41.736677 14358 heartbeater.cc:511] Master 127.10.12.190:44829 requested a full tablet report, sending...
I20260924 03:21:41.738361 13901 ts_manager.cc:194] Registered new tserver with Master: 09fbe424f4b44d468366fdacec5097fa (127.10.12.131:35639)
I20260924 03:21:41.739632 13901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.131@KRBTEST.COM'} at 127.10.12.131:45019
I20260924 03:21:41.744899 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 03:21:41 dist-test-slave-00qs krb5kdc[13856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 03:21:41 dist-test-slave-00qs krb5kdc[13856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.129@KRBTEST.COM
I20260924 03:21:41.791842 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:41.776596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55210 (local address 127.10.12.129:45157)
0924 03:21:41.777280 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:41.777290 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:41.777320 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:21:41.777379 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:41.777385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:41.777618 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:41.777759 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:41.779156 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:41.780214 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:41.781452 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:41.782027 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:41.782144 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:41.785963 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:41.785976 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:41.785991 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:41.786027 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:41.789029 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:41.789721 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:41.789729 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:41.789731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:41.789876 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:41.790223 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:41.790229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:41.790232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:41.790600 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:41.790905 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:41.791212 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 03:21:41.791492 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":344,"threads_started":1}
Sep 24 03:21:41 dist-test-slave-00qs krb5kdc[13856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:41.811291 14226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:41.796622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53778 (local address 127.10.12.190:44829)
0924 03:21:41.796837 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:41.796843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:41.797114 (+ 271us) server_negotiation.cc:407] Connection header received
0924 03:21:41.797181 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:41.797184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:41.797261 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:41.797347 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:41.798524 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:41.799353 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:41.800825 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:41.801229 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:41.801341 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:41.805482 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:41.805489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:41.805492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:41.805535 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:41.808200 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:41.808994 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:41.808999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:41.809001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:41.809156 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:41.809573 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:41.809577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:41.809579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:41.810176 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:41.810407 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:41.810756 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 03:21:41.810989 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260924 03:21:41.811830 13901 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53778
I20260924 03:21:41.812882 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 13962
I20260924 03:21:41.826933 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:41.813943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55220 (local address 127.10.12.129:45157)
0924 03:21:41.814179 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:41.814186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:41.814496 (+ 310us) server_negotiation.cc:407] Connection header received
0924 03:21:41.814694 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:41.814698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:41.814839 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:41.814927 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:41.816192 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:41.816938 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:41.818174 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:41.818591 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:41.818711 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:41.821384 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:41.821390 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:41.821393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:41.821433 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:41.824081 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:41.824827 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:41.824833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:41.824835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:41.824993 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:41.825350 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:41.825355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:41.825357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:41.825781 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:41.826033 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:41.826378 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 03:21:41.826625 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 03:21:41.827569 14085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:41.910992 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 13962
I20260924 03:21:41.924127 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 14096
Sep 24 03:21:41 dist-test-slave-00qs krb5kdc[13856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.130@KRBTEST.COM
I20260924 03:21:41.941738 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:41.925134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58192 (local address 127.10.12.130:42753)
0924 03:21:41.925663 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:41.925670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:41.925698 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:21:41.925771 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:41.925777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:41.925990 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:41.926124 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:41.927456 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:41.928628 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:41.930041 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:41.930613 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:41.930759 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:41.935588 (+ 4829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:41.935604 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:41.935620 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:41.935669 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:41.938558 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:41.939367 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:41.939375 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:41.939377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:41.939521 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:41.939896 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:41.939900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:41.939902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:41.940425 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:41.940744 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:41.941101 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 03:21:41.941394 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":294,"threads_started":1}
I20260924 03:21:41.943284 14219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:42.033838 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 14096
I20260924 03:21:42.046845 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 14230
Sep 24 03:21:42 dist-test-slave-00qs krb5kdc[13856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.131@KRBTEST.COM
I20260924 03:21:42.065141 14372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:42.048154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52524 (local address 127.10.12.131:35639)
0924 03:21:42.048771 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:42.048779 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:42.048811 (+ 32us) server_negotiation.cc:407] Connection header received
0924 03:21:42.048875 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:42.048881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:42.049093 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:42.049228 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:42.050485 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:42.051725 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:42.053348 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:42.054047 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:42.054162 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:42.058775 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:42.058794 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:42.058810 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:42.058856 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:42.062017 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:42.062899 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:42.062907 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:42.062910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:42.063059 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:42.063440 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:42.063444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:42.063446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:42.063829 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:42.064183 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:42.064495 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 03:21:42.064795 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":302,"threads_started":1}
I20260924 03:21:42.066557 14353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:42.149788 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 14230
I20260924 03:21:42.162310 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 13869
I20260924 03:21:42.163705 13931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:42.268095 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 13869
2026-09-24T03:21:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3947 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:21:42 dist-test-slave-00qs krb5kdc[14377](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:21:42 dist-test-slave-00qs krb5kdc[14377](info): set up 2 sockets
Sep 24 03:21:42 dist-test-slave-00qs krb5kdc[14377](info): commencing operation
krb5kdc: starting...
W20260924 03:21:43.289252 10290 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:21:43 dist-test-slave-00qs krb5kdc[14377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:21:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:21:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:21:43.496932 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:44245
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:36769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:44245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:43.833889 14390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:43.834214 14390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:43.834331 14390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:43.841897 14390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:21:43.842031 14390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:43.842082 14390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:43.842159 14390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:21:43.842221 14390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:21:43.852475 14390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:36769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:44245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:44245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:43.853875 14390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:43.855201 14390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:43.867714 14395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:43.867779 14398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:43.867748 14396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:43.869161 14390 server_base.cc:1034] running on GCE node
I20260924 03:21:43.870054 14390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:43.871932 14390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:43.873240 14390 hybrid_clock.cc:648] HybridClock initialized: now 1790220103873192 us; error 43 us; skew 500 ppm
I20260924 03:21:43.873607 14390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:43 dist-test-slave-00qs krb5kdc[14377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220103, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:43.877445 14390 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:21:43.878969 14390 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:43.879084 14390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:43.879386 14390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:43.882688 14390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "0312b824f90a4c31bd7c46c704e78f08"
format_stamp: "Formatted at 2026-09-24 03:21:43 on dist-test-slave-00qs"
server_key: "df302cf589d978d8f19b53a696594558"
server_key_iv: "a63fd47d1110d94f4012ac34cdd2c0ea"
server_key_version: "encryptionkey@0"
I20260924 03:21:43.884372 14390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "0312b824f90a4c31bd7c46c704e78f08"
format_stamp: "Formatted at 2026-09-24 03:21:43 on dist-test-slave-00qs"
server_key: "df302cf589d978d8f19b53a696594558"
server_key_iv: "a63fd47d1110d94f4012ac34cdd2c0ea"
server_key_version: "encryptionkey@0"
I20260924 03:21:43.891693 14390 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260924 03:21:43.896163 14404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:43.898559 14390 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:21:43.898772 14390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "0312b824f90a4c31bd7c46c704e78f08"
format_stamp: "Formatted at 2026-09-24 03:21:43 on dist-test-slave-00qs"
server_key: "df302cf589d978d8f19b53a696594558"
server_key_iv: "a63fd47d1110d94f4012ac34cdd2c0ea"
server_key_version: "encryptionkey@0"
I20260924 03:21:43.899051 14390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:43.914219 14390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:43.918871 14390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:43.919174 14390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:43.938164 14390 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:44245
I20260924 03:21:43.938148 14455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:44245 every 8 connection(s)
I20260924 03:21:43.939127 14390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:21:43.943113 14456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:43.947820 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 14390
I20260924 03:21:43.948117 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:21:43.948922 10290 external_mini_cluster.cc:1428] Setting key f51a06dfa3f352f2dbb1798cbc736f72
I20260924 03:21:43.955430 14456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0312b824f90a4c31bd7c46c704e78f08: Bootstrap starting.
I20260924 03:21:43.958693 14456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0312b824f90a4c31bd7c46c704e78f08: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:43.959740 14456 log.cc:824] T 00000000000000000000000000000000 P 0312b824f90a4c31bd7c46c704e78f08: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:43.964032 14456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0312b824f90a4c31bd7c46c704e78f08: No bootstrap required, opened a new log
Sep 24 03:21:43 dist-test-slave-00qs krb5kdc[14377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:43.973251 14456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0312b824f90a4c31bd7c46c704e78f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0312b824f90a4c31bd7c46c704e78f08" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44245 } }
I20260924 03:21:43.973760 14456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0312b824f90a4c31bd7c46c704e78f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:43.973870 14456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0312b824f90a4c31bd7c46c704e78f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0312b824f90a4c31bd7c46c704e78f08, State: Initialized, Role: FOLLOWER
I20260924 03:21:43.974638 14456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0312b824f90a4c31bd7c46c704e78f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0312b824f90a4c31bd7c46c704e78f08" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44245 } }
I20260924 03:21:43.974938 14456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0312b824f90a4c31bd7c46c704e78f08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:21:43.975121 14456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0312b824f90a4c31bd7c46c704e78f08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:21:43.975157 14459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:43.951631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59690 (local address 127.10.12.190:44245)
0924 03:21:43.952607 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:43.952620 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:43.952677 (+ 57us) server_negotiation.cc:407] Connection header received
0924 03:21:43.954540 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:43.954567 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:43.955017 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:43.955434 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:43.956991 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:43.958338 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:43.959702 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:43.960719 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:43.960950 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:43.966458 (+ 5508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:43.966467 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:43.966486 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:43.966566 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:43.969798 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:43.970821 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:43.970832 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:43.970839 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:43.971094 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:43.971556 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:43.971564 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:43.971566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:43.972340 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:43.972569 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:43.973102 (+ 533us) server_negotiation.cc:299] Negotiation successful
0924 03:21:43.973531 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":485,"threads_started":1}
I20260924 03:21:43.975337 14456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0312b824f90a4c31bd7c46c704e78f08 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:43.978977 14456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0312b824f90a4c31bd7c46c704e78f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0312b824f90a4c31bd7c46c704e78f08" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44245 } }
I20260924 03:21:43.979652 14456 leader_election.cc:302] T 00000000000000000000000000000000 P 0312b824f90a4c31bd7c46c704e78f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0312b824f90a4c31bd7c46c704e78f08; no voters:
I20260924 03:21:43.980563 14456 leader_election.cc:288] T 00000000000000000000000000000000 P 0312b824f90a4c31bd7c46c704e78f08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:21:43.980698 14461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0312b824f90a4c31bd7c46c704e78f08 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:43.983023 14461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0312b824f90a4c31bd7c46c704e78f08 [term 1 LEADER]: Becoming Leader. State: Replica: 0312b824f90a4c31bd7c46c704e78f08, State: Running, Role: LEADER
I20260924 03:21:43.983453 14456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0312b824f90a4c31bd7c46c704e78f08 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:21:43.983688 14461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0312b824f90a4c31bd7c46c704e78f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0312b824f90a4c31bd7c46c704e78f08" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44245 } }
I20260924 03:21:43.987098 14463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0312b824f90a4c31bd7c46c704e78f08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0312b824f90a4c31bd7c46c704e78f08. Latest consensus state: current_term: 1 leader_uuid: "0312b824f90a4c31bd7c46c704e78f08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0312b824f90a4c31bd7c46c704e78f08" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44245 } } }
I20260924 03:21:43.987576 14463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0312b824f90a4c31bd7c46c704e78f08 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:43.987077 14462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0312b824f90a4c31bd7c46c704e78f08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0312b824f90a4c31bd7c46c704e78f08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0312b824f90a4c31bd7c46c704e78f08" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44245 } } }
I20260924 03:21:43.988312 14462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0312b824f90a4c31bd7c46c704e78f08 [sys.catalog]: This master's current role is: LEADER
W20260924 03:21:43.996948 14477 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0312b824f90a4c31bd7c46c704e78f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:21:43.997120 14477 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:21:43.997283 14472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:21:43.999652 14472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:21:44.010933 14472 catalog_manager.cc:1347] Generated new cluster ID: 6a86e7f39c69411fb24bacad923a2caa
I20260924 03:21:44.011066 14472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:21:44.033582 14472 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:21:44.035043 14472 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:21:44.042508 14472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0312b824f90a4c31bd7c46c704e78f08: Generated new TSK 0
I20260924 03:21:44.043529 14472 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
I20260924 03:21:44.097817 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:44245
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:36769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:44.437103 14482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:44.437418 14482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:44.437520 14482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:44.444948 14482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:44.445083 14482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:44.445201 14482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:21:44.455504 14482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:36769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:44245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:44.456939 14482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:44.458273 14482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:44.471505 14488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:44.471583 14487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:44.471563 14490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:44.473186 14482 server_base.cc:1034] running on GCE node
I20260924 03:21:44.473851 14482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:44.474876 14482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:44.476076 14482 hybrid_clock.cc:648] HybridClock initialized: now 1790220104476027 us; error 43 us; skew 500 ppm
I20260924 03:21:44.476464 14482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:44 dist-test-slave-00qs krb5kdc[14377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220104, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:44.479780 14482 init.cc:377] Logged in from keytab as kudu/127.10.12.129@KRBTEST.COM (short username kudu)
I20260924 03:21:44.480960 14482 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:44.481069 14482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:44.481393 14482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:44.484669 14482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "a9b303ace4a94199821cfb519a8b391d"
format_stamp: "Formatted at 2026-09-24 03:21:44 on dist-test-slave-00qs"
server_key: "3328079f9413781c171b9c31490269cf"
server_key_iv: "2d2864578ec6bb662b295b33b1921781"
server_key_version: "encryptionkey@0"
I20260924 03:21:44.485560 14482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "a9b303ace4a94199821cfb519a8b391d"
format_stamp: "Formatted at 2026-09-24 03:21:44 on dist-test-slave-00qs"
server_key: "3328079f9413781c171b9c31490269cf"
server_key_iv: "2d2864578ec6bb662b295b33b1921781"
server_key_version: "encryptionkey@0"
I20260924 03:21:44.491361 14482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:21:44.495527 14496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:44.497210 14482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:21:44.497392 14482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "a9b303ace4a94199821cfb519a8b391d"
format_stamp: "Formatted at 2026-09-24 03:21:44 on dist-test-slave-00qs"
server_key: "3328079f9413781c171b9c31490269cf"
server_key_iv: "2d2864578ec6bb662b295b33b1921781"
server_key_version: "encryptionkey@0"
I20260924 03:21:44.497624 14482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:44.513964 14482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:44.519017 14482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:44.519306 14482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:44.520393 14482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:21:44.521909 14482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:44.522019 14482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:44.522128 14482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:44.522202 14482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:44.553771 14482 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:38795
I20260924 03:21:44.553874 14608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:38795 every 8 connection(s)
I20260924 03:21:44.554670 14482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:21:44.558234 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 14482
I20260924 03:21:44.558470 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:21:44.559320 10290 external_mini_cluster.cc:1428] Setting key 19022db5be3952363d31b61b632843e5
Sep 24 03:21:44 dist-test-slave-00qs krb5kdc[14377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220104, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:44.577222 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:44.557324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:56829 (local address 127.10.12.190:44245)
0924 03:21:44.557791 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:44.557795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:44.558913 (+ 1118us) server_negotiation.cc:407] Connection header received
0924 03:21:44.561047 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:44.561052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:44.561132 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:44.561223 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:44.562920 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:44.563627 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:44.565292 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:44.565772 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:44.565916 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:44.569859 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:44.569866 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:44.569872 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:44.569916 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:44.573242 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:44.574356 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:44.574363 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:44.574366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:44.574547 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:44.575105 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:44.575110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:44.575113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:44.575474 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:44.575629 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:44.576579 (+ 950us) server_negotiation.cc:299] Negotiation successful
0924 03:21:44.576842 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":224,"threads_started":1}
WARNING: no policy specified for kudu/127.10.12.130@KRBTEST.COM; defaulting to no policy
I20260924 03:21:44.578895 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:44.557898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:44245 (local address 127.10.12.129:56829)
0924 03:21:44.558760 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:44.558800 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:44.560652 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:44.561521 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:44.561532 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:44.562081 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:44.562736 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:44.562756 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:44.563787 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:44.563791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:44.565132 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:44.565147 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:44.566004 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:44.566009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:44.566224 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:44.566923 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:44.566952 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:44.569542 (+ 2590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:44.573517 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:44.573530 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:44.573554 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:44.574177 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:44.574741 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:44.574745 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:44.574748 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:44.574978 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:44.575711 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:44.575724 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:44.576162 (+ 438us) client_negotiation.cc:769] Sending connection context
0924 03:21:44.576431 (+ 269us) client_negotiation.cc:240] Negotiation successful
0924 03:21:44.576991 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":393,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.130@KRBTEST.COM" created.
I20260924 03:21:44.581554 14609 heartbeater.cc:347] Connected to a master server at 127.10.12.190:44245
I20260924 03:21:44.581941 14609 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:44.582674 14609 heartbeater.cc:511] Master 127.10.12.190:44245 requested a full tablet report, sending...
I20260924 03:21:44.585089 14421 ts_manager.cc:194] Registered new tserver with Master: a9b303ace4a94199821cfb519a8b391d (127.10.12.129:38795)
I20260924 03:21:44.586649 14421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.129@KRBTEST.COM'} at 127.10.12.129:56829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
I20260924 03:21:44.608528 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:44245
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:36769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:44.943195 14615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:44.943533 14615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:44.943670 14615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:44.951584 14615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:44.951740 14615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:44.951907 14615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:21:44.962375 14615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:36769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:44245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:44.963891 14615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:44.965317 14615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:44.980772 14620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:44.980825 14621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:44.980751 14623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:44.982226 14615 server_base.cc:1034] running on GCE node
I20260924 03:21:44.982975 14615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:44.984185 14615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:44.985528 14615 hybrid_clock.cc:648] HybridClock initialized: now 1790220104985436 us; error 48 us; skew 500 ppm
I20260924 03:21:44.986232 14615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:44 dist-test-slave-00qs krb5kdc[14377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220104, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:44.996182 14615 init.cc:377] Logged in from keytab as kudu/127.10.12.130@KRBTEST.COM (short username kudu)
I20260924 03:21:44.997666 14615 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:44.997802 14615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:44.998149 14615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:45.002451 14615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "555d9fae686742a2aa90d61b15fbbb9c"
format_stamp: "Formatted at 2026-09-24 03:21:44 on dist-test-slave-00qs"
server_key: "50e992f26a945c5fcefe7af4ac629a1f"
server_key_iv: "c22c5030343af6d2149dbf226b361067"
server_key_version: "encryptionkey@0"
I20260924 03:21:45.003515 14615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "555d9fae686742a2aa90d61b15fbbb9c"
format_stamp: "Formatted at 2026-09-24 03:21:44 on dist-test-slave-00qs"
server_key: "50e992f26a945c5fcefe7af4ac629a1f"
server_key_iv: "c22c5030343af6d2149dbf226b361067"
server_key_version: "encryptionkey@0"
I20260924 03:21:45.010075 14615 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20260924 03:21:45.016357 14629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:45.018575 14615 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260924 03:21:45.018864 14615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "555d9fae686742a2aa90d61b15fbbb9c"
format_stamp: "Formatted at 2026-09-24 03:21:44 on dist-test-slave-00qs"
server_key: "50e992f26a945c5fcefe7af4ac629a1f"
server_key_iv: "c22c5030343af6d2149dbf226b361067"
server_key_version: "encryptionkey@0"
I20260924 03:21:45.019184 14615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:45.034590 14615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:45.039242 14615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:45.039534 14615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:45.040566 14615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:21:45.041977 14615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:45.042074 14615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:45.042183 14615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:45.042250 14615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:45.074713 14615 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:45427
I20260924 03:21:45.074748 14741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:45427 every 8 connection(s)
I20260924 03:21:45.075583 14615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:21:45.079751 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 14615
I20260924 03:21:45.080058 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:21:45.080489 10290 external_mini_cluster.cc:1428] Setting key 7ac3b8d840be7675e4d450de8648b035
Sep 24 03:21:45 dist-test-slave-00qs krb5kdc[14377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220104, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:45.096764 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:45.078144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:59607 (local address 127.10.12.190:44245)
0924 03:21:45.078690 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:45.078695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:45.079620 (+ 925us) server_negotiation.cc:407] Connection header received
0924 03:21:45.082094 (+ 2474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:45.082099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:45.082176 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:45.082294 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:45.083920 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.084778 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.086468 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.086901 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.087047 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:45.090944 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:45.090950 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:45.090953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:45.090989 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:45.093571 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:45.094451 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:45.094455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:45.094457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:45.094580 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:45.094995 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:45.095000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:45.095003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:45.095284 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:45.095418 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:45.096157 (+ 739us) server_negotiation.cc:299] Negotiation successful
0924 03:21:45.096390 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":210,"threads_started":1}
WARNING: no policy specified for kudu/127.10.12.131@KRBTEST.COM; defaulting to no policy
I20260924 03:21:45.097797 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:45.078754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:44245 (local address 127.10.12.130:59607)
0924 03:21:45.079456 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:45.079500 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:45.081801 (+ 2301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:45.082522 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:45.082531 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:45.083072 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:45.083746 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:45.083765 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.084950 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.084954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:45.086295 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:45.086310 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.087152 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.087161 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:45.087427 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:45.088214 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:45.088243 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:45.090660 (+ 2417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:45.093753 (+ 3093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:45.093764 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:45.093783 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:45.094310 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:45.094690 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:45.094695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:45.094699 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:45.094890 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:45.095408 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:45.095417 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:45.095781 (+ 364us) client_negotiation.cc:769] Sending connection context
0924 03:21:45.096070 (+ 289us) client_negotiation.cc:240] Negotiation successful
0924 03:21:45.096624 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":264,"threads_started":1}
I20260924 03:21:45.099781 14742 heartbeater.cc:347] Connected to a master server at 127.10.12.190:44245
I20260924 03:21:45.100118 14742 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.131@KRBTEST.COM" created.
I20260924 03:21:45.100889 14742 heartbeater.cc:511] Master 127.10.12.190:44245 requested a full tablet report, sending...
I20260924 03:21:45.103026 14421 ts_manager.cc:194] Registered new tserver with Master: 555d9fae686742a2aa90d61b15fbbb9c (127.10.12.130:45427)
I20260924 03:21:45.104336 14421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.130@KRBTEST.COM'} at 127.10.12.130:59607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
I20260924 03:21:45.124935 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:44245
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:36769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:45.461083 14748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:45.461403 14748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:45.461539 14748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:45.469704 14748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:45.469854 14748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:45.470010 14748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:21:45.480705 14748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:36769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:44245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:45.482142 14748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:45.483502 14748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:45.496901 14754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:45.496937 14756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:45.497164 14753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:45.498164 14748 server_base.cc:1034] running on GCE node
I20260924 03:21:45.498803 14748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:45.499902 14748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:45.501137 14748 hybrid_clock.cc:648] HybridClock initialized: now 1790220105501087 us; error 43 us; skew 500 ppm
I20260924 03:21:45.501466 14748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:45 dist-test-slave-00qs krb5kdc[14377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220105, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:45.505681 14748 init.cc:377] Logged in from keytab as kudu/127.10.12.131@KRBTEST.COM (short username kudu)
I20260924 03:21:45.506969 14748 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:45.507084 14748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:45.507407 14748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:45.510717 14748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "e08d6b9a0ef040cdb82af27ae14c51e2"
format_stamp: "Formatted at 2026-09-24 03:21:45 on dist-test-slave-00qs"
server_key: "90f1e277f7f9cb4a38422939531eff4a"
server_key_iv: "1a94fa159e0c11dd93a5b3662d5aeaec"
server_key_version: "encryptionkey@0"
I20260924 03:21:45.511727 14748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "e08d6b9a0ef040cdb82af27ae14c51e2"
format_stamp: "Formatted at 2026-09-24 03:21:45 on dist-test-slave-00qs"
server_key: "90f1e277f7f9cb4a38422939531eff4a"
server_key_iv: "1a94fa159e0c11dd93a5b3662d5aeaec"
server_key_version: "encryptionkey@0"
I20260924 03:21:45.517464 14748 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260924 03:21:45.522082 14762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:45.523828 14748 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 03:21:45.524073 14748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "e08d6b9a0ef040cdb82af27ae14c51e2"
format_stamp: "Formatted at 2026-09-24 03:21:45 on dist-test-slave-00qs"
server_key: "90f1e277f7f9cb4a38422939531eff4a"
server_key_iv: "1a94fa159e0c11dd93a5b3662d5aeaec"
server_key_version: "encryptionkey@0"
I20260924 03:21:45.524333 14748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:45.560679 14748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:45.565485 14748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:45.565765 14748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:45.566766 14748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:21:45.568285 14748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:45.568387 14748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:45.568498 14748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:45.568580 14748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:45.591243 14609 heartbeater.cc:503] Master 127.10.12.190:44245 was elected leader, sending a full tablet report...
I20260924 03:21:45.600579 14748 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:36623
I20260924 03:21:45.600648 14874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:36623 every 8 connection(s)
I20260924 03:21:45.601456 14748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:21:45.605563 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 14748
I20260924 03:21:45.605770 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:21:45.606302 10290 external_mini_cluster.cc:1428] Setting key badbc85dddd3e160126803137934d560
Sep 24 03:21:45 dist-test-slave-00qs krb5kdc[14377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220105, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:45.622644 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:45.603938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:50599 (local address 127.10.12.190:44245)
0924 03:21:45.604453 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:45.604459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:45.605514 (+ 1055us) server_negotiation.cc:407] Connection header received
0924 03:21:45.608071 (+ 2557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:45.608075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:45.608144 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:45.608240 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:45.610110 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.610889 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.612584 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.613023 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.613135 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:45.616855 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:45.616861 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:45.616865 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:45.616902 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:45.619367 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:45.620261 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:45.620265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:45.620267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:45.620395 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:45.620842 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:45.620846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:45.620848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:45.621117 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:45.621262 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:45.622025 (+ 763us) server_negotiation.cc:299] Negotiation successful
0924 03:21:45.622269 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":220,"threads_started":1}
I20260924 03:21:45.623693 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:45.604552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:44245 (local address 127.10.12.131:50599)
0924 03:21:45.605320 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:45.605370 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:45.607662 (+ 2292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:45.608533 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:45.608547 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:45.609320 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:45.609931 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:45.609951 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.611049 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.611053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:45.612428 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:45.612443 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.613254 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.613262 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:45.613497 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:45.614161 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:45.614186 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:45.616558 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:45.619542 (+ 2984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:45.619551 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:45.619567 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:45.620122 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:45.620506 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:45.620511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:45.620515 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:45.620717 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:45.621248 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:45.621257 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:45.621650 (+ 393us) client_negotiation.cc:769] Sending connection context
0924 03:21:45.621909 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 03:21:45.622470 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":288,"threads_started":1}
I20260924 03:21:45.625342 14875 heartbeater.cc:347] Connected to a master server at 127.10.12.190:44245
I20260924 03:21:45.625674 14875 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:45.626353 14875 heartbeater.cc:511] Master 127.10.12.190:44245 requested a full tablet report, sending...
I20260924 03:21:45.627931 14421 ts_manager.cc:194] Registered new tserver with Master: e08d6b9a0ef040cdb82af27ae14c51e2 (127.10.12.131:36623)
I20260924 03:21:45.628799 14421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.131@KRBTEST.COM'} at 127.10.12.131:50599
I20260924 03:21:45.634255 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:21:45.651818 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:45.638942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59694 (local address 127.10.12.190:44245)
0924 03:21:45.639132 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:45.639136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:45.639553 (+ 417us) server_negotiation.cc:407] Connection header received
0924 03:21:45.639783 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:45.639787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:45.639897 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:45.640053 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:45.641566 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.642356 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.643719 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.644235 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.644340 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:45.646805 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:45.646810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:45.646814 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:45.646852 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:45.649404 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:45.650110 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:45.650114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:45.650116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:45.650241 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:45.650602 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:45.650605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:45.650607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:45.650927 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:45.651111 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:45.651394 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 03:21:45.651588 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 03:21:45.656316 14421 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:21:45.663331 14421 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:21:45.691933 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:45.676270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:36623 (local address 127.0.0.1:43262)
0924 03:21:45.678013 (+ 1743us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:45.678061 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:45.678271 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:45.681481 (+ 3210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:45.681833 (+ 352us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:45.681915 (+ 82us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:45.682633 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:45.682655 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.685312 (+ 2657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.685320 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:45.688347 (+ 3027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:45.688370 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.690771 (+ 2401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.690779 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:45.691033 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:45.691099 (+ 66us) client_negotiation.cc:769] Sending connection context
0924 03:21:45.691229 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 03:21:45.691484 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1426,"thread_start_us":4634,"threads_started":1}
I20260924 03:21:45.693378 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:45.676621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.10.12.131:36623)
0924 03:21:45.679944 (+ 3323us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:45.679982 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:45.680020 (+ 38us) server_negotiation.cc:407] Connection header received
0924 03:21:45.680131 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:45.680139 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:45.680473 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:45.682984 (+ 2511us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:45.683321 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.685107 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.688695 (+ 3588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.690484 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.692391 (+ 1907us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:45.692635 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:45.692734 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 03:21:45.693031 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3057,"thread_start_us":2638,"threads_started":1}
I20260924 03:21:45.693905 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:45.676621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:38795 (local address 127.0.0.1:58898)
0924 03:21:45.681197 (+ 4576us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:45.681219 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:45.681474 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:45.682526 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:45.682532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:45.682569 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:45.683042 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:45.683054 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.687188 (+ 4134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.687194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:45.689765 (+ 2571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:45.689788 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.693051 (+ 3263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.693057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:45.693248 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:45.693279 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 03:21:45.693409 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 03:21:45.693561 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4438,"mutex_wait_us":1,"thread_start_us":918,"threads_started":1}
I20260924 03:21:45.694274 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:45.677291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58898 (local address 127.10.12.129:38795)
0924 03:21:45.678955 (+ 1664us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:45.678966 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:45.681354 (+ 2388us) server_negotiation.cc:407] Connection header received
0924 03:21:45.681665 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:45.681674 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:45.681969 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:45.682169 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:45.683242 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.685096 (+ 1854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.690041 (+ 4945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.691555 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.691675 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:45.691925 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:45.693643 (+ 1718us) server_negotiation.cc:299] Negotiation successful
0924 03:21:45.693952 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1456,"thread_start_us":1300,"threads_started":1}
I20260924 03:21:45.697693 14544 tablet_service.cc:1467] Processing CreateTablet for tablet eb0071cf3ffa467bb350a49536a2a666 (DEFAULT_TABLE table=test-table [id=6c2eb247b8004fcf8e3f8a017174aa7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:45.697912 14810 tablet_service.cc:1467] Processing CreateTablet for tablet eb0071cf3ffa467bb350a49536a2a666 (DEFAULT_TABLE table=test-table [id=6c2eb247b8004fcf8e3f8a017174aa7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:45.699337 14544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb0071cf3ffa467bb350a49536a2a666. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:45.699358 14810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb0071cf3ffa467bb350a49536a2a666. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:45.702641 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:45.680221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:45427 (local address 127.0.0.1:37794)
0924 03:21:45.690357 (+ 10136us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:45.690380 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:45.691584 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:45.694767 (+ 3183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:45.694772 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:45.694805 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:45.695148 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:45.695161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.697257 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.697298 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:45.699583 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:45.699602 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.701625 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.701635 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:45.701821 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:45.701858 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 03:21:45.701963 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 03:21:45.702114 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9987,"thread_start_us":11106,"threads_started":1}
I20260924 03:21:45.702656 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:45.681964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.10.12.130:45427)
0924 03:21:45.692952 (+ 10988us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:45.692965 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:45.692995 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:21:45.693096 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:45.693104 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:45.694526 (+ 1422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:45.694830 (+ 304us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:45.695355 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.697006 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.699803 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.701312 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.701508 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:45.701702 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:45.702029 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 03:21:45.702224 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10804,"thread_start_us":10634,"threads_started":1}
I20260924 03:21:45.707485 14677 tablet_service.cc:1467] Processing CreateTablet for tablet eb0071cf3ffa467bb350a49536a2a666 (DEFAULT_TABLE table=test-table [id=6c2eb247b8004fcf8e3f8a017174aa7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:45.709092 14677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb0071cf3ffa467bb350a49536a2a666. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:45.713333 14894 tablet_bootstrap.cc:492] T eb0071cf3ffa467bb350a49536a2a666 P a9b303ace4a94199821cfb519a8b391d: Bootstrap starting.
I20260924 03:21:45.715528 14895 tablet_bootstrap.cc:492] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2: Bootstrap starting.
I20260924 03:21:45.717964 14894 tablet_bootstrap.cc:654] T eb0071cf3ffa467bb350a49536a2a666 P a9b303ace4a94199821cfb519a8b391d: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:45.719797 14894 log.cc:824] T eb0071cf3ffa467bb350a49536a2a666 P a9b303ace4a94199821cfb519a8b391d: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:45.720544 14895 tablet_bootstrap.cc:654] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:45.722074 14895 log.cc:824] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:45.723896 14898 tablet_bootstrap.cc:492] T eb0071cf3ffa467bb350a49536a2a666 P 555d9fae686742a2aa90d61b15fbbb9c: Bootstrap starting.
I20260924 03:21:45.725132 14894 tablet_bootstrap.cc:492] T eb0071cf3ffa467bb350a49536a2a666 P a9b303ace4a94199821cfb519a8b391d: No bootstrap required, opened a new log
I20260924 03:21:45.725468 14894 ts_tablet_manager.cc:1397] T eb0071cf3ffa467bb350a49536a2a666 P a9b303ace4a94199821cfb519a8b391d: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260924 03:21:45.727654 14898 tablet_bootstrap.cc:654] T eb0071cf3ffa467bb350a49536a2a666 P 555d9fae686742a2aa90d61b15fbbb9c: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:45.729143 14898 log.cc:824] T eb0071cf3ffa467bb350a49536a2a666 P 555d9fae686742a2aa90d61b15fbbb9c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:45.731371 14895 tablet_bootstrap.cc:492] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2: No bootstrap required, opened a new log
I20260924 03:21:45.731709 14895 ts_tablet_manager.cc:1397] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260924 03:21:45.732621 14898 tablet_bootstrap.cc:492] T eb0071cf3ffa467bb350a49536a2a666 P 555d9fae686742a2aa90d61b15fbbb9c: No bootstrap required, opened a new log
I20260924 03:21:45.732892 14898 ts_tablet_manager.cc:1397] T eb0071cf3ffa467bb350a49536a2a666 P 555d9fae686742a2aa90d61b15fbbb9c: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260924 03:21:45.735574 14894 raft_consensus.cc:348] T eb0071cf3ffa467bb350a49536a2a666 P a9b303ace4a94199821cfb519a8b391d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08d6b9a0ef040cdb82af27ae14c51e2" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36623 } } peers { permanent_uuid: "a9b303ace4a94199821cfb519a8b391d" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38795 } } peers { permanent_uuid: "555d9fae686742a2aa90d61b15fbbb9c" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45427 } }
I20260924 03:21:45.736342 14894 raft_consensus.cc:374] T eb0071cf3ffa467bb350a49536a2a666 P a9b303ace4a94199821cfb519a8b391d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:45.736511 14894 raft_consensus.cc:729] T eb0071cf3ffa467bb350a49536a2a666 P a9b303ace4a94199821cfb519a8b391d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9b303ace4a94199821cfb519a8b391d, State: Initialized, Role: FOLLOWER
I20260924 03:21:45.737318 14894 consensus_queue.cc:260] T eb0071cf3ffa467bb350a49536a2a666 P a9b303ace4a94199821cfb519a8b391d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08d6b9a0ef040cdb82af27ae14c51e2" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36623 } } peers { permanent_uuid: "a9b303ace4a94199821cfb519a8b391d" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38795 } } peers { permanent_uuid: "555d9fae686742a2aa90d61b15fbbb9c" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45427 } }
I20260924 03:21:45.739331 14894 ts_tablet_manager.cc:1428] T eb0071cf3ffa467bb350a49536a2a666 P a9b303ace4a94199821cfb519a8b391d: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260924 03:21:45.742944 14895 raft_consensus.cc:348] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08d6b9a0ef040cdb82af27ae14c51e2" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36623 } } peers { permanent_uuid: "a9b303ace4a94199821cfb519a8b391d" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38795 } } peers { permanent_uuid: "555d9fae686742a2aa90d61b15fbbb9c" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45427 } }
I20260924 03:21:45.743525 14895 raft_consensus.cc:374] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:45.743646 14895 raft_consensus.cc:729] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e08d6b9a0ef040cdb82af27ae14c51e2, State: Initialized, Role: FOLLOWER
I20260924 03:21:45.743613 14898 raft_consensus.cc:348] T eb0071cf3ffa467bb350a49536a2a666 P 555d9fae686742a2aa90d61b15fbbb9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08d6b9a0ef040cdb82af27ae14c51e2" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36623 } } peers { permanent_uuid: "a9b303ace4a94199821cfb519a8b391d" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38795 } } peers { permanent_uuid: "555d9fae686742a2aa90d61b15fbbb9c" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45427 } }
I20260924 03:21:45.744067 14898 raft_consensus.cc:374] T eb0071cf3ffa467bb350a49536a2a666 P 555d9fae686742a2aa90d61b15fbbb9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:45.744158 14898 raft_consensus.cc:729] T eb0071cf3ffa467bb350a49536a2a666 P 555d9fae686742a2aa90d61b15fbbb9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555d9fae686742a2aa90d61b15fbbb9c, State: Initialized, Role: FOLLOWER
I20260924 03:21:45.744408 14895 consensus_queue.cc:260] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08d6b9a0ef040cdb82af27ae14c51e2" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36623 } } peers { permanent_uuid: "a9b303ace4a94199821cfb519a8b391d" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38795 } } peers { permanent_uuid: "555d9fae686742a2aa90d61b15fbbb9c" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45427 } }
I20260924 03:21:45.744765 14898 consensus_queue.cc:260] T eb0071cf3ffa467bb350a49536a2a666 P 555d9fae686742a2aa90d61b15fbbb9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08d6b9a0ef040cdb82af27ae14c51e2" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36623 } } peers { permanent_uuid: "a9b303ace4a94199821cfb519a8b391d" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38795 } } peers { permanent_uuid: "555d9fae686742a2aa90d61b15fbbb9c" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45427 } }
I20260924 03:21:45.746503 14742 heartbeater.cc:503] Master 127.10.12.190:44245 was elected leader, sending a full tablet report...
I20260924 03:21:45.747406 14898 ts_tablet_manager.cc:1428] T eb0071cf3ffa467bb350a49536a2a666 P 555d9fae686742a2aa90d61b15fbbb9c: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260924 03:21:45.748497 14875 heartbeater.cc:503] Master 127.10.12.190:44245 was elected leader, sending a full tablet report...
I20260924 03:21:45.748993 14895 ts_tablet_manager.cc:1428] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2: Time spent starting tablet: real 0.017s user 0.009s sys 0.009s
I20260924 03:21:45.768875 14901 raft_consensus.cc:482] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:21:45.769331 14901 raft_consensus.cc:504] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08d6b9a0ef040cdb82af27ae14c51e2" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36623 } } peers { permanent_uuid: "a9b303ace4a94199821cfb519a8b391d" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38795 } } peers { permanent_uuid: "555d9fae686742a2aa90d61b15fbbb9c" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45427 } }
I20260924 03:21:45.771422 14901 leader_election.cc:288] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9b303ace4a94199821cfb519a8b391d (127.10.12.129:38795), 555d9fae686742a2aa90d61b15fbbb9c (127.10.12.130:45427)
I20260924 03:21:45.778075 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:45.771774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:38795 (local address 127.10.12.131:56123)
0924 03:21:45.772204 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:45.772233 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:45.772446 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:45.772863 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:45.772868 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:45.772899 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:45.773250 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:45.773266 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.774411 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.774417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:45.776183 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:45.776196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.777303 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.777309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:45.777521 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:45.777547 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 03:21:45.777634 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 03:21:45.777779 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":4}
I20260924 03:21:45.778059 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:45.772212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:56123 (local address 127.10.12.129:38795)
0924 03:21:45.772477 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:45.772483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:45.772573 (+ 90us) server_negotiation.cc:407] Connection header received
0924 03:21:45.772662 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:45.772666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:45.772738 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:45.772816 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:45.773496 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.774257 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.776359 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.777137 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.777313 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:45.777407 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:45.777679 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 03:21:45.777813 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260924 03:21:45.779294 14564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0071cf3ffa467bb350a49536a2a666" candidate_uuid: "e08d6b9a0ef040cdb82af27ae14c51e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9b303ace4a94199821cfb519a8b391d" is_pre_election: true
I20260924 03:21:45.779626 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:45.772447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:38697 (local address 127.10.12.130:45427)
0924 03:21:45.772627 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:45.772633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:45.773303 (+ 670us) server_negotiation.cc:407] Connection header received
0924 03:21:45.773476 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:45.773483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:45.773568 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:45.773664 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:21:45.774456 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.775574 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.777818 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.778720 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.778886 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:45.778956 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:45.779236 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 03:21:45.779370 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 03:21:45.779690 14903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:45.772235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:45427 (local address 127.10.12.131:38697)
0924 03:21:45.773157 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:45.773176 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:45.773332 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:45.773703 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:45.773707 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:45.773733 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:21:45.774192 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:45.774204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.775743 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.775748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:45.777469 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:45.777557 (+ 88us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.778924 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.778931 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:45.779084 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:45.779107 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 03:21:45.779189 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 03:21:45.779377 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":400,"threads_started":1}
I20260924 03:21:45.779878 14564 raft_consensus.cc:2454] T eb0071cf3ffa467bb350a49536a2a666 P a9b303ace4a94199821cfb519a8b391d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e08d6b9a0ef040cdb82af27ae14c51e2 in term 0.
I20260924 03:21:45.780578 14766 leader_election.cc:302] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9b303ace4a94199821cfb519a8b391d, e08d6b9a0ef040cdb82af27ae14c51e2; no voters:
I20260924 03:21:45.780721 14697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0071cf3ffa467bb350a49536a2a666" candidate_uuid: "e08d6b9a0ef040cdb82af27ae14c51e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555d9fae686742a2aa90d61b15fbbb9c" is_pre_election: true
I20260924 03:21:45.781033 14901 raft_consensus.cc:2781] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:21:45.781175 14901 raft_consensus.cc:482] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:21:45.781271 14901 raft_consensus.cc:3037] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:45.781167 14697 raft_consensus.cc:2454] T eb0071cf3ffa467bb350a49536a2a666 P 555d9fae686742a2aa90d61b15fbbb9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e08d6b9a0ef040cdb82af27ae14c51e2 in term 0.
I20260924 03:21:45.783696 14901 raft_consensus.cc:504] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08d6b9a0ef040cdb82af27ae14c51e2" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36623 } } peers { permanent_uuid: "a9b303ace4a94199821cfb519a8b391d" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38795 } } peers { permanent_uuid: "555d9fae686742a2aa90d61b15fbbb9c" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45427 } }
I20260924 03:21:45.784379 14901 leader_election.cc:288] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2 [CANDIDATE]: Term 1 election: Requested vote from peers a9b303ace4a94199821cfb519a8b391d (127.10.12.129:38795), 555d9fae686742a2aa90d61b15fbbb9c (127.10.12.130:45427)
I20260924 03:21:45.785249 14564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0071cf3ffa467bb350a49536a2a666" candidate_uuid: "e08d6b9a0ef040cdb82af27ae14c51e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9b303ace4a94199821cfb519a8b391d"
I20260924 03:21:45.785473 14564 raft_consensus.cc:3037] T eb0071cf3ffa467bb350a49536a2a666 P a9b303ace4a94199821cfb519a8b391d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:45.785607 14697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0071cf3ffa467bb350a49536a2a666" candidate_uuid: "e08d6b9a0ef040cdb82af27ae14c51e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555d9fae686742a2aa90d61b15fbbb9c"
I20260924 03:21:45.786005 14697 raft_consensus.cc:3037] T eb0071cf3ffa467bb350a49536a2a666 P 555d9fae686742a2aa90d61b15fbbb9c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:45.787880 14564 raft_consensus.cc:2454] T eb0071cf3ffa467bb350a49536a2a666 P a9b303ace4a94199821cfb519a8b391d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e08d6b9a0ef040cdb82af27ae14c51e2 in term 1.
I20260924 03:21:45.788410 14766 leader_election.cc:302] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9b303ace4a94199821cfb519a8b391d, e08d6b9a0ef040cdb82af27ae14c51e2; no voters:
I20260924 03:21:45.788540 14697 raft_consensus.cc:2454] T eb0071cf3ffa467bb350a49536a2a666 P 555d9fae686742a2aa90d61b15fbbb9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e08d6b9a0ef040cdb82af27ae14c51e2 in term 1.
I20260924 03:21:45.788777 14901 raft_consensus.cc:2781] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:45.789248 14901 raft_consensus.cc:686] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2 [term 1 LEADER]: Becoming Leader. State: Replica: e08d6b9a0ef040cdb82af27ae14c51e2, State: Running, Role: LEADER
I20260924 03:21:45.789733 14901 consensus_queue.cc:237] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08d6b9a0ef040cdb82af27ae14c51e2" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36623 } } peers { permanent_uuid: "a9b303ace4a94199821cfb519a8b391d" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38795 } } peers { permanent_uuid: "555d9fae686742a2aa90d61b15fbbb9c" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45427 } }
I20260924 03:21:45.795782 14420 catalog_manager.cc:5495] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e08d6b9a0ef040cdb82af27ae14c51e2 (127.10.12.131). New cstate: current_term: 1 leader_uuid: "e08d6b9a0ef040cdb82af27ae14c51e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08d6b9a0ef040cdb82af27ae14c51e2" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9b303ace4a94199821cfb519a8b391d" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "555d9fae686742a2aa90d61b15fbbb9c" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45427 } health_report { overall_health: UNKNOWN } } }
W20260924 03:21:45.806944 14610 tablet.cc:2369] T eb0071cf3ffa467bb350a49536a2a666 P a9b303ace4a94199821cfb519a8b391d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:21:45.821244 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:45.814946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43266 (local address 127.10.12.131:36623)
0924 03:21:45.815183 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:45.815190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:45.815267 (+ 77us) server_negotiation.cc:407] Connection header received
0924 03:21:45.815463 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:45.815467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:45.815546 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:45.815678 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:45.816283 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.817321 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.818788 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.819267 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.819371 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:45.820050 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:45.820602 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:45.820719 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:45.820894 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 03:21:45.820973 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 03:21:45.827451 14743 tablet.cc:2369] T eb0071cf3ffa467bb350a49536a2a666 P 555d9fae686742a2aa90d61b15fbbb9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:21:45.830693 14564 raft_consensus.cc:1261] T eb0071cf3ffa467bb350a49536a2a666 P a9b303ace4a94199821cfb519a8b391d [term 1 FOLLOWER]: Refusing update from remote peer e08d6b9a0ef040cdb82af27ae14c51e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:21:45.830783 14697 raft_consensus.cc:1261] T eb0071cf3ffa467bb350a49536a2a666 P 555d9fae686742a2aa90d61b15fbbb9c [term 1 FOLLOWER]: Refusing update from remote peer e08d6b9a0ef040cdb82af27ae14c51e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:21:45.831779 14904 consensus_queue.cc:1035] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "555d9fae686742a2aa90d61b15fbbb9c" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:21:45.832195 14901 consensus_queue.cc:1035] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9b303ace4a94199821cfb519a8b391d" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:21:45.849051 14909 mvcc.cc:205] Tried to move back new op lower bound from 7332741553470005248 to 7332741553319100416. Current Snapshot: MvccSnapshot[applied={T|T < 7332741553470005248}]
I20260924 03:21:45.859236 14421 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:21:45.859565 14421 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:21:45.864063 14421 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0312b824f90a4c31bd7c46c704e78f08: Sending DeleteTablet for 3 replicas of tablet eb0071cf3ffa467bb350a49536a2a666
I20260924 03:21:45.865526 14677 tablet_service.cc:1514] Processing DeleteTablet for tablet eb0071cf3ffa467bb350a49536a2a666 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:21:45 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:37794
I20260924 03:21:45.865708 14810 tablet_service.cc:1514] Processing DeleteTablet for tablet eb0071cf3ffa467bb350a49536a2a666 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:21:45 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:43262
I20260924 03:21:45.866307 14544 tablet_service.cc:1514] Processing DeleteTablet for tablet eb0071cf3ffa467bb350a49536a2a666 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:21:45 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:58898
I20260924 03:21:45.866966 14543 tablet_service.cc:1514] Processing DeleteTablet for tablet eb0071cf3ffa467bb350a49536a2a666 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:21:45 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:58898
I20260924 03:21:45.867414 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 14482
I20260924 03:21:45.891197 14919 tablet_replica.cc:330] T eb0071cf3ffa467bb350a49536a2a666 P a9b303ace4a94199821cfb519a8b391d: stopping tablet replica
I20260924 03:21:45.892406 14919 raft_consensus.cc:2229] T eb0071cf3ffa467bb350a49536a2a666 P a9b303ace4a94199821cfb519a8b391d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:21:45.893013 14919 raft_consensus.cc:2258] T eb0071cf3ffa467bb350a49536a2a666 P a9b303ace4a94199821cfb519a8b391d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:21:45.893893 14918 tablet_replica.cc:330] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2: stopping tablet replica
I20260924 03:21:45.894441 14918 raft_consensus.cc:2229] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:21:45.895236 14918 raft_consensus.cc:2258] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:21:45.895941 14919 ts_tablet_manager.cc:1905] T eb0071cf3ffa467bb350a49536a2a666 P a9b303ace4a94199821cfb519a8b391d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:21:45.898630 14918 ts_tablet_manager.cc:1905] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:21:45.902143 14917 tablet_replica.cc:330] T eb0071cf3ffa467bb350a49536a2a666 P 555d9fae686742a2aa90d61b15fbbb9c: stopping tablet replica
I20260924 03:21:45.903553 14917 raft_consensus.cc:2229] T eb0071cf3ffa467bb350a49536a2a666 P 555d9fae686742a2aa90d61b15fbbb9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:21:45.904220 14917 raft_consensus.cc:2258] T eb0071cf3ffa467bb350a49536a2a666 P 555d9fae686742a2aa90d61b15fbbb9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:21:45.904649 14919 ts_tablet_manager.cc:1918] T eb0071cf3ffa467bb350a49536a2a666 P a9b303ace4a94199821cfb519a8b391d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:21:45.904991 14919 log.cc:1196] T eb0071cf3ffa467bb350a49536a2a666 P a9b303ace4a94199821cfb519a8b391d: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-0/wal/wals/eb0071cf3ffa467bb350a49536a2a666
I20260924 03:21:45.905624 14919 ts_tablet_manager.cc:1939] T eb0071cf3ffa467bb350a49536a2a666 P a9b303ace4a94199821cfb519a8b391d: Deleting consensus metadata
I20260924 03:21:45.906754 14917 ts_tablet_manager.cc:1905] T eb0071cf3ffa467bb350a49536a2a666 P 555d9fae686742a2aa90d61b15fbbb9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:21:45.907897 14408 catalog_manager.cc:4841] TS a9b303ace4a94199821cfb519a8b391d (127.10.12.129:38795): tablet eb0071cf3ffa467bb350a49536a2a666 (table test-table [id=6c2eb247b8004fcf8e3f8a017174aa7c]) successfully deleted
W20260924 03:21:45.909147 14408 catalog_manager.cc:4808] TS a9b303ace4a94199821cfb519a8b391d (127.10.12.129:38795): delete failed for tablet eb0071cf3ffa467bb350a49536a2a666 because the tablet was not found. No further retry: Not found: Tablet not found: eb0071cf3ffa467bb350a49536a2a666
Sep 24 03:21:45 dist-test-slave-00qs krb5kdc[14377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.129@KRBTEST.COM
I20260924 03:21:45.914326 14917 ts_tablet_manager.cc:1918] T eb0071cf3ffa467bb350a49536a2a666 P 555d9fae686742a2aa90d61b15fbbb9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:21:45.914535 14917 log.cc:1196] T eb0071cf3ffa467bb350a49536a2a666 P 555d9fae686742a2aa90d61b15fbbb9c: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/eb0071cf3ffa467bb350a49536a2a666
I20260924 03:21:45.915078 14917 ts_tablet_manager.cc:1939] T eb0071cf3ffa467bb350a49536a2a666 P 555d9fae686742a2aa90d61b15fbbb9c: Deleting consensus metadata
I20260924 03:21:45.915712 14918 ts_tablet_manager.cc:1918] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:21:45.915998 14918 log.cc:1196] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/eb0071cf3ffa467bb350a49536a2a666
I20260924 03:21:45.916664 14918 ts_tablet_manager.cc:1939] T eb0071cf3ffa467bb350a49536a2a666 P e08d6b9a0ef040cdb82af27ae14c51e2: Deleting consensus metadata
I20260924 03:21:45.917220 14408 catalog_manager.cc:4841] TS 555d9fae686742a2aa90d61b15fbbb9c (127.10.12.130:45427): tablet eb0071cf3ffa467bb350a49536a2a666 (table test-table [id=6c2eb247b8004fcf8e3f8a017174aa7c]) successfully deleted
I20260924 03:21:45.918730 14405 catalog_manager.cc:4841] TS e08d6b9a0ef040cdb82af27ae14c51e2 (127.10.12.131:36623): tablet eb0071cf3ffa467bb350a49536a2a666 (table test-table [id=6c2eb247b8004fcf8e3f8a017174aa7c]) successfully deleted
I20260924 03:21:45.921775 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:45.868340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58900 (local address 127.10.12.129:38795)
0924 03:21:45.868552 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:45.868559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:45.903255 (+ 34696us) server_negotiation.cc:407] Connection header received
0924 03:21:45.903462 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:45.903468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:45.903554 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:45.903659 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:45.905600 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.906748 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.908618 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:45.909343 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:45.909452 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:45.915050 (+ 5598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:45.915074 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:45.915095 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:45.915154 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:45.918706 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:45.919724 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:45.919733 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:45.919735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:45.919899 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:45.920366 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:45.920369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:45.920371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:45.920749 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:45.921038 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:45.921301 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 03:21:45.921495 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 03:21:45.922466 14604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:46.046973 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 14482
I20260924 03:21:46.061112 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 14615
Sep 24 03:21:46 dist-test-slave-00qs krb5kdc[14377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.130@KRBTEST.COM
I20260924 03:21:46.079468 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:46.062060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37796 (local address 127.10.12.130:45427)
0924 03:21:46.062225 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:46.062229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:46.062248 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:21:46.062447 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:46.062452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:46.062529 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:46.062611 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:46.064494 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:46.065329 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:46.067259 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:46.067731 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:46.067844 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:46.073316 (+ 5472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:46.073336 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:46.073352 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:46.073395 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:46.076513 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:46.077294 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:46.077302 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:46.077304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:46.077441 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:46.077812 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:46.077816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:46.077819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:46.078248 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:46.078636 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:46.078853 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 03:21:46.079103 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 03:21:46.080209 14737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:46.172078 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 14615
I20260924 03:21:46.192358 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 14748
Sep 24 03:21:46 dist-test-slave-00qs krb5kdc[14377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.131@KRBTEST.COM
I20260924 03:21:46.212059 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:46.193404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.10.12.131:36623)
0924 03:21:46.193658 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:46.193664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:46.193693 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:21:46.193772 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:46.193778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:46.193866 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:46.194060 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:46.195677 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:46.197127 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:46.198631 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:46.199392 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:46.199552 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:46.204266 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:46.204290 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:46.204318 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:46.204387 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:46.208761 (+ 4374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:46.209601 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:46.209613 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:46.209615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:46.209796 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:46.210252 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:46.210256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:46.210258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:46.210677 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:46.211011 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:46.211360 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 03:21:46.211640 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260924 03:21:46.212877 14870 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:46.340184 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 14748
I20260924 03:21:46.355258 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 14390
I20260924 03:21:46.356544 14451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:46.466562 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 14390
2026-09-24T03:21:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4197 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T03:21:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:21:46Z Disabled control of system clock
I20260924 03:21:46.530471 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:41185
--webserver_interface=127.10.12.190
--webserver_port=0
--builtin_ntp_servers=127.10.12.148:44369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:41185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 03:21:46.863502 14928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:46.863835 14928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:46.863946 14928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:46.871584 14928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:21:46.871726 14928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:46.871766 14928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:46.871830 14928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:21:46.871903 14928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:21:46.882071 14928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:44369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:41185
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:41185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:46.883426 14928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:46.885510 14928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:46.898418 14933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:46.899328 14934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:46.899379 14936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:46.899678 14928 server_base.cc:1034] running on GCE node
I20260924 03:21:46.900851 14928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:46.902567 14928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:46.906075 14928 hybrid_clock.cc:648] HybridClock initialized: now 1790220106905975 us; error 73 us; skew 500 ppm
I20260924 03:21:46.906554 14928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:21:46.909821 14928 webserver.cc:460] Webserver started at http://127.10.12.190:34965/ using document root <none> and password file <none>
I20260924 03:21:46.910710 14928 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:46.910857 14928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:46.911207 14928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:46.914429 14928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "c726e2eb06b74d1daf5acc8a5eaba63c"
format_stamp: "Formatted at 2026-09-24 03:21:46 on dist-test-slave-00qs"
server_key: "595509404de2d27e04c1ae0310416123"
server_key_iv: "ae21b20212140f3cfb38cfef35f74d33"
server_key_version: "encryptionkey@0"
I20260924 03:21:46.915311 14928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "c726e2eb06b74d1daf5acc8a5eaba63c"
format_stamp: "Formatted at 2026-09-24 03:21:46 on dist-test-slave-00qs"
server_key: "595509404de2d27e04c1ae0310416123"
server_key_iv: "ae21b20212140f3cfb38cfef35f74d33"
server_key_version: "encryptionkey@0"
I20260924 03:21:46.922155 14928 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260924 03:21:46.926934 14942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:46.928612 14928 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:21:46.928850 14928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "c726e2eb06b74d1daf5acc8a5eaba63c"
format_stamp: "Formatted at 2026-09-24 03:21:46 on dist-test-slave-00qs"
server_key: "595509404de2d27e04c1ae0310416123"
server_key_iv: "ae21b20212140f3cfb38cfef35f74d33"
server_key_version: "encryptionkey@0"
I20260924 03:21:46.929081 14928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:46.935484 14928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:46.936447 14928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:46.936743 14928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:46.955206 14928 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:41185
I20260924 03:21:46.955224 14993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:41185 every 8 connection(s)
I20260924 03:21:46.956620 14928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:21:46.960700 14994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:46.960887 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 14928
I20260924 03:21:46.961071 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:21:46.961884 10290 external_mini_cluster.cc:1428] Setting key 737f236a67c8f8542eeb84293a6b4b09
I20260924 03:21:46.973325 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:46.964702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.10.12.190:41185)
0924 03:21:46.966157 (+ 1455us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:46.966177 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:46.966245 (+ 68us) server_negotiation.cc:407] Connection header received
0924 03:21:46.968687 (+ 2442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:46.968723 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:46.969433 (+ 710us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:46.969992 (+ 559us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:46.971276 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:46.971283 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:46.971296 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:21:46.971311 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:46.971373 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:21:46.971389 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:46.971572 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:46.971679 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 03:21:46.972130 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":810,"thread_start_us":2597,"threads_started":1}
I20260924 03:21:46.977818 14994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c726e2eb06b74d1daf5acc8a5eaba63c: Bootstrap starting.
I20260924 03:21:46.981077 14994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c726e2eb06b74d1daf5acc8a5eaba63c: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:46.982223 14994 log.cc:824] T 00000000000000000000000000000000 P c726e2eb06b74d1daf5acc8a5eaba63c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:46.985710 14994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c726e2eb06b74d1daf5acc8a5eaba63c: No bootstrap required, opened a new log
I20260924 03:21:46.994729 14994 raft_consensus.cc:348] T 00000000000000000000000000000000 P c726e2eb06b74d1daf5acc8a5eaba63c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c726e2eb06b74d1daf5acc8a5eaba63c" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 41185 } }
I20260924 03:21:46.995160 14994 raft_consensus.cc:374] T 00000000000000000000000000000000 P c726e2eb06b74d1daf5acc8a5eaba63c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:46.995272 14994 raft_consensus.cc:729] T 00000000000000000000000000000000 P c726e2eb06b74d1daf5acc8a5eaba63c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c726e2eb06b74d1daf5acc8a5eaba63c, State: Initialized, Role: FOLLOWER
I20260924 03:21:46.995916 14994 consensus_queue.cc:260] T 00000000000000000000000000000000 P c726e2eb06b74d1daf5acc8a5eaba63c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c726e2eb06b74d1daf5acc8a5eaba63c" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 41185 } }
I20260924 03:21:46.996186 14994 raft_consensus.cc:388] T 00000000000000000000000000000000 P c726e2eb06b74d1daf5acc8a5eaba63c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:21:46.996321 14994 raft_consensus.cc:482] T 00000000000000000000000000000000 P c726e2eb06b74d1daf5acc8a5eaba63c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:21:46.996503 14994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c726e2eb06b74d1daf5acc8a5eaba63c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:46.998775 14994 raft_consensus.cc:504] T 00000000000000000000000000000000 P c726e2eb06b74d1daf5acc8a5eaba63c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c726e2eb06b74d1daf5acc8a5eaba63c" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 41185 } }
I20260924 03:21:46.999384 14994 leader_election.cc:302] T 00000000000000000000000000000000 P c726e2eb06b74d1daf5acc8a5eaba63c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c726e2eb06b74d1daf5acc8a5eaba63c; no voters:
I20260924 03:21:47.000167 14994 leader_election.cc:288] T 00000000000000000000000000000000 P c726e2eb06b74d1daf5acc8a5eaba63c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:21:47.000289 14999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c726e2eb06b74d1daf5acc8a5eaba63c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:47.000947 14999 raft_consensus.cc:686] T 00000000000000000000000000000000 P c726e2eb06b74d1daf5acc8a5eaba63c [term 1 LEADER]: Becoming Leader. State: Replica: c726e2eb06b74d1daf5acc8a5eaba63c, State: Running, Role: LEADER
I20260924 03:21:47.001498 14999 consensus_queue.cc:237] T 00000000000000000000000000000000 P c726e2eb06b74d1daf5acc8a5eaba63c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c726e2eb06b74d1daf5acc8a5eaba63c" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 41185 } }
I20260924 03:21:47.002303 14994 sys_catalog.cc:564] T 00000000000000000000000000000000 P c726e2eb06b74d1daf5acc8a5eaba63c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:21:47.005287 15000 sys_catalog.cc:455] T 00000000000000000000000000000000 P c726e2eb06b74d1daf5acc8a5eaba63c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c726e2eb06b74d1daf5acc8a5eaba63c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c726e2eb06b74d1daf5acc8a5eaba63c" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 41185 } } }
I20260924 03:21:47.005316 15001 sys_catalog.cc:455] T 00000000000000000000000000000000 P c726e2eb06b74d1daf5acc8a5eaba63c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c726e2eb06b74d1daf5acc8a5eaba63c. Latest consensus state: current_term: 1 leader_uuid: "c726e2eb06b74d1daf5acc8a5eaba63c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c726e2eb06b74d1daf5acc8a5eaba63c" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 41185 } } }
I20260924 03:21:47.006459 15001 sys_catalog.cc:458] T 00000000000000000000000000000000 P c726e2eb06b74d1daf5acc8a5eaba63c [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:47.006182 15000 sys_catalog.cc:458] T 00000000000000000000000000000000 P c726e2eb06b74d1daf5acc8a5eaba63c [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:47.008349 15006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:21:47.013625 15006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:21:47.024793 15006 catalog_manager.cc:1347] Generated new cluster ID: 95a03889d2d04b9e8456a228c173d98a
I20260924 03:21:47.024935 15006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:21:47.041544 15006 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:21:47.042127 15006 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:21:47.057452 15006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c726e2eb06b74d1daf5acc8a5eaba63c: Generated new TSK 0
I20260924 03:21:47.058357 15006 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:21:47.068293 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:41185
--builtin_ntp_servers=127.10.12.148:44369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 03:21:47.425871 15018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:47.426352 15018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:47.426509 15018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:47.435286 15018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:47.435437 15018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:47.435599 15018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:21:47.446587 15018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:44369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:41185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:47.448035 15018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:47.449911 15018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:47.466059 15024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:47.466248 15023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:47.466065 15026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:47.467351 15018 server_base.cc:1034] running on GCE node
I20260924 03:21:47.468225 15018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:47.469432 15018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:47.470711 15018 hybrid_clock.cc:648] HybridClock initialized: now 1790220107470641 us; error 64 us; skew 500 ppm
I20260924 03:21:47.471098 15018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:21:47.473878 15018 webserver.cc:460] Webserver started at http://127.10.12.129:40477/ using document root <none> and password file <none>
I20260924 03:21:47.474661 15018 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:47.474784 15018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:47.475152 15018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:47.478717 15018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "4ee78e7d76b14a5583c3234545adb712"
format_stamp: "Formatted at 2026-09-24 03:21:47 on dist-test-slave-00qs"
server_key: "2247abb15487e5ea36e010a589010183"
server_key_iv: "e5179b998d0c061ed1458446304c3e8d"
server_key_version: "encryptionkey@0"
I20260924 03:21:47.479892 15018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "4ee78e7d76b14a5583c3234545adb712"
format_stamp: "Formatted at 2026-09-24 03:21:47 on dist-test-slave-00qs"
server_key: "2247abb15487e5ea36e010a589010183"
server_key_iv: "e5179b998d0c061ed1458446304c3e8d"
server_key_version: "encryptionkey@0"
I20260924 03:21:47.487051 15018 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260924 03:21:47.491703 15032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:47.494225 15018 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260924 03:21:47.494463 15018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "4ee78e7d76b14a5583c3234545adb712"
format_stamp: "Formatted at 2026-09-24 03:21:47 on dist-test-slave-00qs"
server_key: "2247abb15487e5ea36e010a589010183"
server_key_iv: "e5179b998d0c061ed1458446304c3e8d"
server_key_version: "encryptionkey@0"
I20260924 03:21:47.494868 15018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:47.502944 15018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:47.504069 15018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:47.504351 15018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:47.505364 15018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:21:47.506965 15018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:47.507066 15018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:47.507156 15018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:47.507228 15018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:47.541517 15018 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:33651
I20260924 03:21:47.541599 15144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:33651 every 8 connection(s)
I20260924 03:21:47.542893 15018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:21:47.550151 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 15018
I20260924 03:21:47.550439 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:21:47.551302 10290 external_mini_cluster.cc:1428] Setting key 086d819b7eadcfc01cca3a8fa32b2ba9
I20260924 03:21:47.552666 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:47.545841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:47551 (local address 127.10.12.190:41185)
0924 03:21:47.546445 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:47.546452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:47.547433 (+ 981us) server_negotiation.cc:407] Connection header received
0924 03:21:47.549368 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:47.549376 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:47.549504 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:47.549769 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:47.551573 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:47.551579 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:47.551583 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:21:47.551604 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:47.551656 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:21:47.551672 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:47.551798 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:47.552222 (+ 424us) server_negotiation.cc:299] Negotiation successful
0924 03:21:47.552328 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":294,"threads_started":1}
I20260924 03:21:47.553721 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:47.546178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:41185 (local address 127.10.12.129:47551)
0924 03:21:47.547265 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:47.547308 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:47.548932 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:47.550138 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:47.550151 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:47.550306 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:47.551137 (+ 831us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:21:47.551140 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:47.551195 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:21:47.551327 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:47.551801 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:47.551812 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:47.551820 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 03:21:47.552275 (+ 455us) client_negotiation.cc:240] Negotiation successful
0924 03:21:47.552588 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":333,"threads_started":1}
I20260924 03:21:47.554662 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:41185
--builtin_ntp_servers=127.10.12.148:44369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 03:21:47.555606 15145 heartbeater.cc:347] Connected to a master server at 127.10.12.190:41185
I20260924 03:21:47.556176 15145 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:47.556849 15145 heartbeater.cc:511] Master 127.10.12.190:41185 requested a full tablet report, sending...
I20260924 03:21:47.559239 14959 ts_manager.cc:194] Registered new tserver with Master: 4ee78e7d76b14a5583c3234545adb712 (127.10.12.129:33651)
W20260924 03:21:47.898696 15149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:47.899024 15149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:47.899142 15149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:47.907006 15149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:47.907147 15149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:47.907305 15149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:21:47.917171 15149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:44369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:41185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:47.918545 15149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:47.920595 15149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:47.936589 15155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:47.936744 15157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:47.936744 15154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:47.938124 15149 server_base.cc:1034] running on GCE node
I20260924 03:21:47.938897 15149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:47.940168 15149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:47.941423 15149 hybrid_clock.cc:648] HybridClock initialized: now 1790220107941360 us; error 63 us; skew 500 ppm
I20260924 03:21:47.941792 15149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:21:47.944631 15149 webserver.cc:460] Webserver started at http://127.10.12.130:44885/ using document root <none> and password file <none>
I20260924 03:21:47.945420 15149 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:47.945535 15149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:47.945905 15149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:47.949432 15149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "83c0351656044f998fb261567118aa34"
format_stamp: "Formatted at 2026-09-24 03:21:47 on dist-test-slave-00qs"
server_key: "35e56460261d4ada70b094ff9e1c402b"
server_key_iv: "49359234ab618b19fa352af88d1cfd6d"
server_key_version: "encryptionkey@0"
I20260924 03:21:47.950791 15149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "83c0351656044f998fb261567118aa34"
format_stamp: "Formatted at 2026-09-24 03:21:47 on dist-test-slave-00qs"
server_key: "35e56460261d4ada70b094ff9e1c402b"
server_key_iv: "49359234ab618b19fa352af88d1cfd6d"
server_key_version: "encryptionkey@0"
I20260924 03:21:47.957916 15149 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260924 03:21:47.962034 15163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:47.964274 15149 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:21:47.964496 15149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "83c0351656044f998fb261567118aa34"
format_stamp: "Formatted at 2026-09-24 03:21:47 on dist-test-slave-00qs"
server_key: "35e56460261d4ada70b094ff9e1c402b"
server_key_iv: "49359234ab618b19fa352af88d1cfd6d"
server_key_version: "encryptionkey@0"
I20260924 03:21:47.964741 15149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:47.971052 15149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:47.971927 15149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:47.972230 15149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:47.973258 15149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:21:47.975203 15149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:47.975322 15149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:47.975440 15149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:47.975512 15149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:48.022681 15149 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:35187
I20260924 03:21:48.022735 15275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:35187 every 8 connection(s)
I20260924 03:21:48.024154 15149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:21:48.027767 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 15149
I20260924 03:21:48.027910 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:21:48.028398 10290 external_mini_cluster.cc:1428] Setting key 1fcf4e4a0c3760f05a9abed5b4366a01
I20260924 03:21:48.031812 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:41185
--builtin_ntp_servers=127.10.12.148:44369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 03:21:48.035496 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:48.027159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:55731 (local address 127.10.12.190:41185)
0924 03:21:48.027373 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:48.027381 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:48.030442 (+ 3061us) server_negotiation.cc:407] Connection header received
0924 03:21:48.032988 (+ 2546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:48.032993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:48.033070 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:48.033197 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:48.034773 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:48.034778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:48.034781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:21:48.034801 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:48.034834 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:21:48.034846 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:48.034962 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:48.035191 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 03:21:48.035251 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 03:21:48.036819 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:48.027641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:41185 (local address 127.10.12.130:55731)
0924 03:21:48.030283 (+ 2642us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:48.030341 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:48.032605 (+ 2264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:48.033460 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:48.033473 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:48.033614 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:48.034404 (+ 790us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:21:48.034408 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:48.034462 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:21:48.034583 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:48.034943 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:48.034952 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:48.034959 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 03:21:48.035235 (+ 276us) client_negotiation.cc:240] Negotiation successful
0924 03:21:48.035540 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2088,"thread_start_us":1904,"threads_started":1}
I20260924 03:21:48.038424 15276 heartbeater.cc:347] Connected to a master server at 127.10.12.190:41185
I20260924 03:21:48.038769 15276 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:48.039400 15276 heartbeater.cc:511] Master 127.10.12.190:41185 requested a full tablet report, sending...
I20260924 03:21:48.040920 14959 ts_manager.cc:194] Registered new tserver with Master: 83c0351656044f998fb261567118aa34 (127.10.12.130:35187)
W20260924 03:21:48.398293 15279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:48.398684 15279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:48.398830 15279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:48.408176 15279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:48.408319 15279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:48.408496 15279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:21:48.420272 15279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:44369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:41185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:48.421837 15279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:48.423846 15279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:48.439464 15287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:48.439497 15284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:48.439862 15285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:48.441391 15279 server_base.cc:1034] running on GCE node
I20260924 03:21:48.442493 15279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:48.443687 15279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:48.445088 15279 hybrid_clock.cc:648] HybridClock initialized: now 1790220108444963 us; error 113 us; skew 500 ppm
I20260924 03:21:48.445641 15279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:21:48.448459 15279 webserver.cc:460] Webserver started at http://127.10.12.131:38227/ using document root <none> and password file <none>
I20260924 03:21:48.449265 15279 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:48.449388 15279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:48.449741 15279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:48.453261 15279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "ae41ee5c188a45d08b6b8a316c65da57"
format_stamp: "Formatted at 2026-09-24 03:21:48 on dist-test-slave-00qs"
server_key: "5c33bf94f464121f05cfb12322efb024"
server_key_iv: "ad2e4bf290cd08ac75ffad8143c9b3cf"
server_key_version: "encryptionkey@0"
I20260924 03:21:48.454350 15279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "ae41ee5c188a45d08b6b8a316c65da57"
format_stamp: "Formatted at 2026-09-24 03:21:48 on dist-test-slave-00qs"
server_key: "5c33bf94f464121f05cfb12322efb024"
server_key_iv: "ad2e4bf290cd08ac75ffad8143c9b3cf"
server_key_version: "encryptionkey@0"
I20260924 03:21:48.462132 15279 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260924 03:21:48.466517 15293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:48.468979 15279 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 03:21:48.469199 15279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "ae41ee5c188a45d08b6b8a316c65da57"
format_stamp: "Formatted at 2026-09-24 03:21:48 on dist-test-slave-00qs"
server_key: "5c33bf94f464121f05cfb12322efb024"
server_key_iv: "ad2e4bf290cd08ac75ffad8143c9b3cf"
server_key_version: "encryptionkey@0"
I20260924 03:21:48.469455 15279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:48.478343 15279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:48.479383 15279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:48.479787 15279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:48.480998 15279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:21:48.482641 15279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:48.482753 15279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:48.482884 15279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:48.482954 15279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:48.523417 15279 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:39803
I20260924 03:21:48.523485 15405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:39803 every 8 connection(s)
I20260924 03:21:48.524879 15279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:21:48.526239 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 15279
I20260924 03:21:48.526387 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:21:48.526894 10290 external_mini_cluster.cc:1428] Setting key 761995bede4e38352fe59b0908c59a0e
I20260924 03:21:48.535498 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:48.528765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:52099 (local address 127.10.12.190:41185)
0924 03:21:48.528971 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:48.528978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:48.531042 (+ 2064us) server_negotiation.cc:407] Connection header received
0924 03:21:48.532890 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:48.532896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:48.532974 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:48.533083 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:48.534741 (+ 1658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:48.534746 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:48.534749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:21:48.534768 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:48.534803 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:21:48.534816 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:48.534972 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:48.535182 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 03:21:48.535243 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 03:21:48.536810 15408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:48.529544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:41185 (local address 127.10.12.131:52099)
0924 03:21:48.530850 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:48.530904 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:48.532544 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:48.533328 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:48.533339 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:48.533444 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:48.534143 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:21:48.534145 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:48.534187 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:21:48.534286 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:48.534951 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:48.534960 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:48.534966 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 03:21:48.535204 (+ 238us) client_negotiation.cc:240] Negotiation successful
0924 03:21:48.535449 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":459,"threads_started":1}
I20260924 03:21:48.538440 15406 heartbeater.cc:347] Connected to a master server at 127.10.12.190:41185
I20260924 03:21:48.538744 15406 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:48.539378 15406 heartbeater.cc:511] Master 127.10.12.190:41185 requested a full tablet report, sending...
I20260924 03:21:48.544863 14959 ts_manager.cc:194] Registered new tserver with Master: ae41ee5c188a45d08b6b8a316c65da57 (127.10.12.131:39803)
I20260924 03:21:48.556339 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:21:48.563886 15145 heartbeater.cc:503] Master 127.10.12.190:41185 was elected leader, sending a full tablet report...
I20260924 03:21:48.565779 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:48.561740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60414 (local address 127.10.12.190:41185)
0924 03:21:48.562070 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:48.562077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:48.562802 (+ 725us) server_negotiation.cc:407] Connection header received
0924 03:21:48.563012 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:48.563018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:48.563098 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:48.563248 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:48.565067 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:48.565073 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:48.565076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:21:48.565098 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:48.565136 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:21:48.565149 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:48.565252 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:48.565400 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 03:21:48.565454 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260924 03:21:48.571709 14959 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:21:48.580632 14959 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:21:48.614148 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:48.603044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37328 (local address 127.10.12.131:39803)
0924 03:21:48.611267 (+ 8223us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:48.611277 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:48.611321 (+ 44us) server_negotiation.cc:407] Connection header received
0924 03:21:48.611405 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:48.611414 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:48.611680 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:48.611876 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:48.613300 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:48.613308 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:48.613323 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:21:48.613340 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:48.613397 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:21:48.613417 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:48.613592 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:48.613663 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 03:21:48.613767 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8045,"thread_start_us":7870,"threads_started":1}
I20260924 03:21:48.614197 15422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:48.602715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:39803 (local address 127.0.0.1:37328)
0924 03:21:48.610790 (+ 8075us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:48.610808 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:48.610909 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:48.611909 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:48.611918 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:48.612048 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:48.612866 (+ 818us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:21:48.612870 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:48.612905 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:21:48.612993 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:48.613517 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:48.613526 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:48.613533 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 03:21:48.613666 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 03:21:48.613739 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7902,"thread_start_us":7743,"threads_started":1}
I20260924 03:21:48.616194 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:48.600396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54756 (local address 127.10.12.130:35187)
0924 03:21:48.611360 (+ 10964us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:48.611368 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:48.611398 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:21:48.611478 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:48.611486 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:48.611732 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:48.611902 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:48.615207 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:48.615218 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:48.615233 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:21:48.615267 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:48.615330 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:21:48.615353 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:48.615494 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:48.615721 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 03:21:48.615826 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10804,"spinlock_wait_cycles":24960,"thread_start_us":10782,"threads_started":1}
I20260924 03:21:48.616276 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:48.600378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:33651 (local address 127.0.0.1:60590)
0924 03:21:48.610480 (+ 10102us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:48.610502 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:48.610628 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:48.613939 (+ 3311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:48.613943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:48.613968 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:48.614654 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:21:48.614657 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:48.614673 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:21:48.614718 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:48.615567 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:48.615573 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:48.615578 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:21:48.615724 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 03:21:48.615782 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9919,"thread_start_us":8659,"threads_started":1}
I20260924 03:21:48.616521 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:48.601199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60590 (local address 127.10.12.129:33651)
0924 03:21:48.613366 (+ 12167us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:48.613375 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:48.613405 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:21:48.613485 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:48.613493 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:48.613739 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:48.613907 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:48.615324 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:48.615344 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:48.615356 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:21:48.615373 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:48.615423 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:21:48.615440 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:48.616026 (+ 586us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:48.616092 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 03:21:48.616179 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11988,"thread_start_us":11367,"threads_started":1}
I20260924 03:21:48.616801 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:48.600151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:35187 (local address 127.0.0.1:54756)
0924 03:21:48.609588 (+ 9437us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:48.609651 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:48.609847 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:48.612953 (+ 3106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:48.612958 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:48.612983 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:48.614962 (+ 1979us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:21:48.614965 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:48.614979 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:21:48.615015 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:48.615548 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:48.615553 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:48.615558 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:21:48.615746 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 03:21:48.615800 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9107,"thread_start_us":9964,"threads_started":1}
I20260924 03:21:48.618590 15341 tablet_service.cc:1467] Processing CreateTablet for tablet 1b20ac6d34114df2926aad184e21e04d (DEFAULT_TABLE table=test-table [id=7d7eb9f042a542f7951fc1d79a1b35c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:48.620258 15341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b20ac6d34114df2926aad184e21e04d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:48.620381 15211 tablet_service.cc:1467] Processing CreateTablet for tablet 1b20ac6d34114df2926aad184e21e04d (DEFAULT_TABLE table=test-table [id=7d7eb9f042a542f7951fc1d79a1b35c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:48.620685 15080 tablet_service.cc:1467] Processing CreateTablet for tablet 1b20ac6d34114df2926aad184e21e04d (DEFAULT_TABLE table=test-table [id=7d7eb9f042a542f7951fc1d79a1b35c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:21:48.621892 15211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b20ac6d34114df2926aad184e21e04d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:48.621912 15080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b20ac6d34114df2926aad184e21e04d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:48.632462 15424 tablet_bootstrap.cc:492] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57: Bootstrap starting.
I20260924 03:21:48.636080 15424 tablet_bootstrap.cc:654] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:48.637416 15425 tablet_bootstrap.cc:492] T 1b20ac6d34114df2926aad184e21e04d P 83c0351656044f998fb261567118aa34: Bootstrap starting.
I20260924 03:21:48.637369 15424 log.cc:824] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:48.638427 15426 tablet_bootstrap.cc:492] T 1b20ac6d34114df2926aad184e21e04d P 4ee78e7d76b14a5583c3234545adb712: Bootstrap starting.
I20260924 03:21:48.642055 15426 tablet_bootstrap.cc:654] T 1b20ac6d34114df2926aad184e21e04d P 4ee78e7d76b14a5583c3234545adb712: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:48.642234 15425 tablet_bootstrap.cc:654] T 1b20ac6d34114df2926aad184e21e04d P 83c0351656044f998fb261567118aa34: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:48.643143 15426 log.cc:824] T 1b20ac6d34114df2926aad184e21e04d P 4ee78e7d76b14a5583c3234545adb712: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:48.643196 15424 tablet_bootstrap.cc:492] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57: No bootstrap required, opened a new log
I20260924 03:21:48.643498 15424 ts_tablet_manager.cc:1397] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260924 03:21:48.643457 15425 log.cc:824] T 1b20ac6d34114df2926aad184e21e04d P 83c0351656044f998fb261567118aa34: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:48.646206 15425 tablet_bootstrap.cc:492] T 1b20ac6d34114df2926aad184e21e04d P 83c0351656044f998fb261567118aa34: No bootstrap required, opened a new log
I20260924 03:21:48.646390 15426 tablet_bootstrap.cc:492] T 1b20ac6d34114df2926aad184e21e04d P 4ee78e7d76b14a5583c3234545adb712: No bootstrap required, opened a new log
I20260924 03:21:48.646513 15425 ts_tablet_manager.cc:1397] T 1b20ac6d34114df2926aad184e21e04d P 83c0351656044f998fb261567118aa34: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260924 03:21:48.646692 15426 ts_tablet_manager.cc:1397] T 1b20ac6d34114df2926aad184e21e04d P 4ee78e7d76b14a5583c3234545adb712: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 03:21:48.654330 15425 raft_consensus.cc:348] T 1b20ac6d34114df2926aad184e21e04d P 83c0351656044f998fb261567118aa34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0351656044f998fb261567118aa34" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 35187 } } peers { permanent_uuid: "4ee78e7d76b14a5583c3234545adb712" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 33651 } } peers { permanent_uuid: "ae41ee5c188a45d08b6b8a316c65da57" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39803 } }
I20260924 03:21:48.654352 15424 raft_consensus.cc:348] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0351656044f998fb261567118aa34" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 35187 } } peers { permanent_uuid: "4ee78e7d76b14a5583c3234545adb712" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 33651 } } peers { permanent_uuid: "ae41ee5c188a45d08b6b8a316c65da57" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39803 } }
I20260924 03:21:48.654773 15425 raft_consensus.cc:374] T 1b20ac6d34114df2926aad184e21e04d P 83c0351656044f998fb261567118aa34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:48.654794 15424 raft_consensus.cc:374] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:48.654888 15424 raft_consensus.cc:729] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae41ee5c188a45d08b6b8a316c65da57, State: Initialized, Role: FOLLOWER
I20260924 03:21:48.654919 15425 raft_consensus.cc:729] T 1b20ac6d34114df2926aad184e21e04d P 83c0351656044f998fb261567118aa34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c0351656044f998fb261567118aa34, State: Initialized, Role: FOLLOWER
I20260924 03:21:48.655570 15425 consensus_queue.cc:260] T 1b20ac6d34114df2926aad184e21e04d P 83c0351656044f998fb261567118aa34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0351656044f998fb261567118aa34" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 35187 } } peers { permanent_uuid: "4ee78e7d76b14a5583c3234545adb712" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 33651 } } peers { permanent_uuid: "ae41ee5c188a45d08b6b8a316c65da57" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39803 } }
I20260924 03:21:48.655586 15424 consensus_queue.cc:260] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0351656044f998fb261567118aa34" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 35187 } } peers { permanent_uuid: "4ee78e7d76b14a5583c3234545adb712" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 33651 } } peers { permanent_uuid: "ae41ee5c188a45d08b6b8a316c65da57" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39803 } }
I20260924 03:21:48.656831 15276 heartbeater.cc:503] Master 127.10.12.190:41185 was elected leader, sending a full tablet report...
I20260924 03:21:48.657523 15425 ts_tablet_manager.cc:1428] T 1b20ac6d34114df2926aad184e21e04d P 83c0351656044f998fb261567118aa34: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260924 03:21:48.657650 15426 raft_consensus.cc:348] T 1b20ac6d34114df2926aad184e21e04d P 4ee78e7d76b14a5583c3234545adb712 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0351656044f998fb261567118aa34" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 35187 } } peers { permanent_uuid: "4ee78e7d76b14a5583c3234545adb712" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 33651 } } peers { permanent_uuid: "ae41ee5c188a45d08b6b8a316c65da57" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39803 } }
I20260924 03:21:48.658188 15426 raft_consensus.cc:374] T 1b20ac6d34114df2926aad184e21e04d P 4ee78e7d76b14a5583c3234545adb712 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:48.658336 15426 raft_consensus.cc:729] T 1b20ac6d34114df2926aad184e21e04d P 4ee78e7d76b14a5583c3234545adb712 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee78e7d76b14a5583c3234545adb712, State: Initialized, Role: FOLLOWER
I20260924 03:21:48.659276 15426 consensus_queue.cc:260] T 1b20ac6d34114df2926aad184e21e04d P 4ee78e7d76b14a5583c3234545adb712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0351656044f998fb261567118aa34" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 35187 } } peers { permanent_uuid: "4ee78e7d76b14a5583c3234545adb712" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 33651 } } peers { permanent_uuid: "ae41ee5c188a45d08b6b8a316c65da57" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39803 } }
I20260924 03:21:48.664072 15424 ts_tablet_manager.cc:1428] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57: Time spent starting tablet: real 0.020s user 0.009s sys 0.014s
I20260924 03:21:48.664464 15431 raft_consensus.cc:482] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:21:48.664599 15406 heartbeater.cc:503] Master 127.10.12.190:41185 was elected leader, sending a full tablet report...
I20260924 03:21:48.664755 15431 raft_consensus.cc:504] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0351656044f998fb261567118aa34" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 35187 } } peers { permanent_uuid: "4ee78e7d76b14a5583c3234545adb712" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 33651 } } peers { permanent_uuid: "ae41ee5c188a45d08b6b8a316c65da57" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39803 } }
I20260924 03:21:48.665356 15426 ts_tablet_manager.cc:1428] T 1b20ac6d34114df2926aad184e21e04d P 4ee78e7d76b14a5583c3234545adb712: Time spent starting tablet: real 0.019s user 0.013s sys 0.004s
I20260924 03:21:48.667209 15431 leader_election.cc:288] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83c0351656044f998fb261567118aa34 (127.10.12.130:35187), 4ee78e7d76b14a5583c3234545adb712 (127.10.12.129:33651)
I20260924 03:21:48.670929 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:48.667909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:48893 (local address 127.10.12.130:35187)
0924 03:21:48.668130 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:48.668136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:48.668158 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:21:48.668327 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:48.668334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:48.668398 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:48.668486 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:48.670286 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:48.670292 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:48.670297 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:21:48.670318 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:48.670358 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:21:48.670386 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:48.670504 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:48.670602 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 03:21:48.670665 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260924 03:21:48.670933 15408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:48.667382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:35187 (local address 127.10.12.131:48893)
0924 03:21:48.668030 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:48.668051 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:48.668163 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:48.668670 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:48.668675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:48.668708 (+ 33us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:48.669999 (+ 1291us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:21:48.670002 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:48.670020 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:21:48.670071 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:48.670474 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:48.670478 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:48.670484 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 03:21:48.670593 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 03:21:48.670651 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493}
I20260924 03:21:48.672171 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:48.668097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:52431 (local address 127.10.12.129:33651)
0924 03:21:48.668450 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:48.668457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:48.668965 (+ 508us) server_negotiation.cc:407] Connection header received
0924 03:21:48.669604 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:48.669610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:48.669666 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:48.669761 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:48.671414 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:48.671420 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:48.671424 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:21:48.671444 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:48.671483 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:21:48.671499 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:48.671596 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:48.671828 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 03:21:48.671891 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239}
I20260924 03:21:48.672308 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:48.668265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:33651 (local address 127.10.12.131:52431)
0924 03:21:48.668858 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:48.668887 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:48.669479 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:48.669845 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:48.669849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:48.669873 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:48.671171 (+ 1298us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:21:48.671174 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:48.671190 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:21:48.671237 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:48.671678 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:48.671683 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:48.671687 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:21:48.671842 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 03:21:48.671891 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":315,"threads_started":1}
I20260924 03:21:48.672072 15231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b20ac6d34114df2926aad184e21e04d" candidate_uuid: "ae41ee5c188a45d08b6b8a316c65da57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c0351656044f998fb261567118aa34" is_pre_election: true
I20260924 03:21:48.672691 15231 raft_consensus.cc:2454] T 1b20ac6d34114df2926aad184e21e04d P 83c0351656044f998fb261567118aa34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae41ee5c188a45d08b6b8a316c65da57 in term 0.
I20260924 03:21:48.673120 15100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b20ac6d34114df2926aad184e21e04d" candidate_uuid: "ae41ee5c188a45d08b6b8a316c65da57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee78e7d76b14a5583c3234545adb712" is_pre_election: true
I20260924 03:21:48.673415 15295 leader_election.cc:302] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83c0351656044f998fb261567118aa34, ae41ee5c188a45d08b6b8a316c65da57; no voters:
I20260924 03:21:48.673635 15100 raft_consensus.cc:2454] T 1b20ac6d34114df2926aad184e21e04d P 4ee78e7d76b14a5583c3234545adb712 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae41ee5c188a45d08b6b8a316c65da57 in term 0.
I20260924 03:21:48.673858 15431 raft_consensus.cc:2781] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:21:48.674008 15431 raft_consensus.cc:482] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:21:48.674118 15431 raft_consensus.cc:3037] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:48.676460 15431 raft_consensus.cc:504] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0351656044f998fb261567118aa34" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 35187 } } peers { permanent_uuid: "4ee78e7d76b14a5583c3234545adb712" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 33651 } } peers { permanent_uuid: "ae41ee5c188a45d08b6b8a316c65da57" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39803 } }
I20260924 03:21:48.677038 15431 leader_election.cc:288] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57 [CANDIDATE]: Term 1 election: Requested vote from peers 83c0351656044f998fb261567118aa34 (127.10.12.130:35187), 4ee78e7d76b14a5583c3234545adb712 (127.10.12.129:33651)
I20260924 03:21:48.678012 15100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b20ac6d34114df2926aad184e21e04d" candidate_uuid: "ae41ee5c188a45d08b6b8a316c65da57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee78e7d76b14a5583c3234545adb712"
I20260924 03:21:48.678012 15231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b20ac6d34114df2926aad184e21e04d" candidate_uuid: "ae41ee5c188a45d08b6b8a316c65da57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c0351656044f998fb261567118aa34"
I20260924 03:21:48.678380 15231 raft_consensus.cc:3037] T 1b20ac6d34114df2926aad184e21e04d P 83c0351656044f998fb261567118aa34 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:48.678380 15100 raft_consensus.cc:3037] T 1b20ac6d34114df2926aad184e21e04d P 4ee78e7d76b14a5583c3234545adb712 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:48.682394 15231 raft_consensus.cc:2454] T 1b20ac6d34114df2926aad184e21e04d P 83c0351656044f998fb261567118aa34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae41ee5c188a45d08b6b8a316c65da57 in term 1.
I20260924 03:21:48.682399 15100 raft_consensus.cc:2454] T 1b20ac6d34114df2926aad184e21e04d P 4ee78e7d76b14a5583c3234545adb712 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae41ee5c188a45d08b6b8a316c65da57 in term 1.
I20260924 03:21:48.683359 15295 leader_election.cc:302] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4ee78e7d76b14a5583c3234545adb712, 83c0351656044f998fb261567118aa34, ae41ee5c188a45d08b6b8a316c65da57; no voters:
I20260924 03:21:48.683846 15431 raft_consensus.cc:2781] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:48.684156 15431 raft_consensus.cc:686] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57 [term 1 LEADER]: Becoming Leader. State: Replica: ae41ee5c188a45d08b6b8a316c65da57, State: Running, Role: LEADER
I20260924 03:21:48.684723 15431 consensus_queue.cc:237] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0351656044f998fb261567118aa34" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 35187 } } peers { permanent_uuid: "4ee78e7d76b14a5583c3234545adb712" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 33651 } } peers { permanent_uuid: "ae41ee5c188a45d08b6b8a316c65da57" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39803 } }
I20260924 03:21:48.690014 14959 catalog_manager.cc:5495] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae41ee5c188a45d08b6b8a316c65da57 (127.10.12.131). New cstate: current_term: 1 leader_uuid: "ae41ee5c188a45d08b6b8a316c65da57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0351656044f998fb261567118aa34" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 35187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ee78e7d76b14a5583c3234545adb712" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 33651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae41ee5c188a45d08b6b8a316c65da57" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39803 } health_report { overall_health: HEALTHY } } }
I20260924 03:21:48.720582 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:48.718362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37336 (local address 127.10.12.131:39803)
0924 03:21:48.718564 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:48.718568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:48.718611 (+ 43us) server_negotiation.cc:407] Connection header received
0924 03:21:48.718711 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:48.718719 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:48.718811 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:48.718896 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:48.719982 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:48.719990 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:48.719994 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:21:48.720016 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:48.720050 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:21:48.720064 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:48.720170 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:48.720273 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 03:21:48.720332 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 03:21:48.729357 15100 raft_consensus.cc:1261] T 1b20ac6d34114df2926aad184e21e04d P 4ee78e7d76b14a5583c3234545adb712 [term 1 FOLLOWER]: Refusing update from remote peer ae41ee5c188a45d08b6b8a316c65da57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:21:48.729928 15231 raft_consensus.cc:1261] T 1b20ac6d34114df2926aad184e21e04d P 83c0351656044f998fb261567118aa34 [term 1 FOLLOWER]: Refusing update from remote peer ae41ee5c188a45d08b6b8a316c65da57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:21:48.730706 15432 consensus_queue.cc:1035] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ee78e7d76b14a5583c3234545adb712" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 33651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:21:48.731129 15431 consensus_queue.cc:1035] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c0351656044f998fb261567118aa34" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:21:48.742715 15437 mvcc.cc:205] Tried to move back new op lower bound from 7332741565344780288 to 7332741565176438784. Current Snapshot: MvccSnapshot[applied={T|T < 7332741565344780288}]
I20260924 03:21:48.761706 14959 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:21:48.762091 14959 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:21:48.766605 14959 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c726e2eb06b74d1daf5acc8a5eaba63c: Sending DeleteTablet for 3 replicas of tablet 1b20ac6d34114df2926aad184e21e04d
I20260924 03:21:48.768091 15341 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b20ac6d34114df2926aad184e21e04d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:21:48 UTC) from {username='slave'} at 127.0.0.1:37328
I20260924 03:21:48.768085 15080 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b20ac6d34114df2926aad184e21e04d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:21:48 UTC) from {username='slave'} at 127.0.0.1:60590
I20260924 03:21:48.768121 15211 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b20ac6d34114df2926aad184e21e04d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:21:48 UTC) from {username='slave'} at 127.0.0.1:54756
I20260924 03:21:48.770596 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 15018
I20260924 03:21:48.782155 15449 tablet_replica.cc:330] T 1b20ac6d34114df2926aad184e21e04d P 83c0351656044f998fb261567118aa34: stopping tablet replica
I20260924 03:21:48.782595 15449 raft_consensus.cc:2229] T 1b20ac6d34114df2926aad184e21e04d P 83c0351656044f998fb261567118aa34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:21:48.783049 15449 raft_consensus.cc:2258] T 1b20ac6d34114df2926aad184e21e04d P 83c0351656044f998fb261567118aa34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:21:48.784466 15447 tablet_replica.cc:330] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57: stopping tablet replica
I20260924 03:21:48.785102 15447 raft_consensus.cc:2229] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:21:48.785862 15447 raft_consensus.cc:2258] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:21:48.786118 15449 ts_tablet_manager.cc:1905] T 1b20ac6d34114df2926aad184e21e04d P 83c0351656044f998fb261567118aa34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:21:48.786979 15448 tablet_replica.cc:330] T 1b20ac6d34114df2926aad184e21e04d P 4ee78e7d76b14a5583c3234545adb712: stopping tablet replica
I20260924 03:21:48.787406 15448 raft_consensus.cc:2229] T 1b20ac6d34114df2926aad184e21e04d P 4ee78e7d76b14a5583c3234545adb712 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:21:48.787935 15448 raft_consensus.cc:2258] T 1b20ac6d34114df2926aad184e21e04d P 4ee78e7d76b14a5583c3234545adb712 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:21:48.788105 15447 ts_tablet_manager.cc:1905] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:21:48.790372 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:48.771899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60596 (local address 127.10.12.129:33651)
0924 03:21:48.772085 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:48.772091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:48.787580 (+ 15489us) server_negotiation.cc:407] Connection header received
0924 03:21:48.788154 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:48.788161 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:48.788281 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:48.788360 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:48.789556 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:48.789562 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:48.789567 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:21:48.789587 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:48.789629 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:21:48.789644 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:48.789756 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:48.789993 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 03:21:48.790058 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 03:21:48.790657 15448 ts_tablet_manager.cc:1905] T 1b20ac6d34114df2926aad184e21e04d P 4ee78e7d76b14a5583c3234545adb712: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:21:48.791486 15140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:48.797654 15449 ts_tablet_manager.cc:1918] T 1b20ac6d34114df2926aad184e21e04d P 83c0351656044f998fb261567118aa34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:21:48.797858 15449 log.cc:1196] T 1b20ac6d34114df2926aad184e21e04d P 83c0351656044f998fb261567118aa34: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/1b20ac6d34114df2926aad184e21e04d
I20260924 03:21:48.798303 15449 ts_tablet_manager.cc:1939] T 1b20ac6d34114df2926aad184e21e04d P 83c0351656044f998fb261567118aa34: Deleting consensus metadata
I20260924 03:21:48.800163 14944 catalog_manager.cc:4841] TS 83c0351656044f998fb261567118aa34 (127.10.12.130:35187): tablet 1b20ac6d34114df2926aad184e21e04d (table test-table [id=7d7eb9f042a542f7951fc1d79a1b35c8]) successfully deleted
I20260924 03:21:48.801342 15447 ts_tablet_manager.cc:1918] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:21:48.801517 15447 log.cc:1196] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/1b20ac6d34114df2926aad184e21e04d
I20260924 03:21:48.801992 15447 ts_tablet_manager.cc:1939] T 1b20ac6d34114df2926aad184e21e04d P ae41ee5c188a45d08b6b8a316c65da57: Deleting consensus metadata
I20260924 03:21:48.803643 14945 catalog_manager.cc:4841] TS ae41ee5c188a45d08b6b8a316c65da57 (127.10.12.131:39803): tablet 1b20ac6d34114df2926aad184e21e04d (table test-table [id=7d7eb9f042a542f7951fc1d79a1b35c8]) successfully deleted
I20260924 03:21:48.930233 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 15018
I20260924 03:21:48.944607 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 15149
W20260924 03:21:48.944635 14946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.12.129:33651 (error 108)
I20260924 03:21:48.946717 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:48.946217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:33651 (local address 127.0.0.1:60608)
0924 03:21:48.946467 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:48.946576 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260924 03:21:48.946987 14946 catalog_manager.cc:4584] TS 4ee78e7d76b14a5583c3234545adb712 (127.10.12.129:33651): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b20ac6d34114df2926aad184e21e04d: Network error: Client connection negotiation failed: client connection to 127.10.12.129:33651: connect: Connection refused (error 111)
I20260924 03:21:48.948119 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:48.945768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54766 (local address 127.10.12.130:35187)
0924 03:21:48.945958 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:48.945964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:48.945984 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:21:48.946051 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:48.946057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:48.946165 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:48.946270 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:48.947439 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:48.947443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:48.947446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:21:48.947464 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:48.947496 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:21:48.947507 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:48.947620 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:48.947725 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 03:21:48.947775 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 03:21:48.948822 15271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:49.000890 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:49.000284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:33651 (local address 127.0.0.1:60612)
0924 03:21:49.000556 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:49.000678 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260924 03:21:49.002261 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:49.001752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:33651 (local address 127.0.0.1:60628)
0924 03:21:49.001970 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:49.002100 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 03:21:49.051712 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 15149
I20260924 03:21:49.063261 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:49.062535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:33651 (local address 127.0.0.1:60630)
0924 03:21:49.062802 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:49.063067 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260924 03:21:49.064519 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:49.064072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:33651 (local address 127.0.0.1:60640)
0924 03:21:49.064254 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:49.064354 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 03:21:49.065272 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 15279
I20260924 03:21:49.068171 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:49.066102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37348 (local address 127.10.12.131:39803)
0924 03:21:49.066286 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:49.066291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:49.066311 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:21:49.066410 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:49.066416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:49.066476 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:49.066557 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:49.067485 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:49.067491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:49.067496 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:21:49.067515 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:49.067546 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:21:49.067557 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:49.067683 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:49.067779 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 03:21:49.067842 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 03:21:49.068957 15401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:49.145787 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:49.144915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:33651 (local address 127.0.0.1:60656)
0924 03:21:49.145396 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:49.145562 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227}
I20260924 03:21:49.147640 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:49.146985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:33651 (local address 127.0.0.1:60668)
0924 03:21:49.147337 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:49.147458 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20260924 03:21:49.195473 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 15279
I20260924 03:21:49.211760 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 14928
I20260924 03:21:49.213143 14989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:49.324442 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:49.323730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:33651 (local address 127.0.0.1:60676)
0924 03:21:49.324122 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:49.324222 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239}
I20260924 03:21:49.324674 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 14928
2026-09-24T03:21:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2857 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 03:21:49.362658 10290 mini_oidc.cc:152] Starting JWKS server
I20260924 03:21:49.365278 10290 webserver.cc:460] Webserver started at https://127.0.0.1:45577/ using document root <none> and password file <none>
I20260924 03:21:49.365569 10290 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 03:21:49.366207 10290 webserver.cc:460] Webserver started at http://127.0.0.1:42135/ using document root <none> and password file <none>
2026-09-24T03:21:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:21:49Z Disabled control of system clock
I20260924 03:21:49.394712 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:34671
--webserver_interface=127.10.12.190
--webserver_port=0
--builtin_ntp_servers=127.10.12.148:40983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:34671
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45577/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790220067268388-10290-0/testchainca.cert with env {}
W20260924 03:21:49.741474 15461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:49.741814 15461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:49.741938 15461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:49.749464 15461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:21:49.749600 15461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:49.749637 15461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:49.749699 15461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:21:49.749769 15461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 03:21:49.749845 15461 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 03:21:49.749893 15461 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45577/jwks.json
I20260924 03:21:49.760458 15461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:40983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:34671
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:34671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45577/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790220067268388-10290-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:49.761863 15461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:49.763252 15461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:49.776891 15466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:49.777202 15469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:49.777249 15467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:49.777978 15461 server_base.cc:1034] running on GCE node
I20260924 03:21:49.778813 15461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:49.780329 15461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:49.781539 15461 hybrid_clock.cc:648] HybridClock initialized: now 1790220109781488 us; error 46 us; skew 500 ppm
I20260924 03:21:49.781913 15461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:21:49.784374 15461 webserver.cc:460] Webserver started at http://127.10.12.190:33473/ using document root <none> and password file <none>
I20260924 03:21:49.785130 15461 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:49.785224 15461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:49.785514 15461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:49.788602 15461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "5a5cf3ad69f34a808ac1302ba0c12850"
format_stamp: "Formatted at 2026-09-24 03:21:49 on dist-test-slave-00qs"
server_key: "dfcd343d33ac26da71eb510682702df6"
server_key_iv: "dd57ce6f0e8978d8c00d12019e3842d6"
server_key_version: "encryptionkey@0"
I20260924 03:21:49.789462 15461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "5a5cf3ad69f34a808ac1302ba0c12850"
format_stamp: "Formatted at 2026-09-24 03:21:49 on dist-test-slave-00qs"
server_key: "dfcd343d33ac26da71eb510682702df6"
server_key_iv: "dd57ce6f0e8978d8c00d12019e3842d6"
server_key_version: "encryptionkey@0"
I20260924 03:21:49.795203 15461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:21:49.799137 15475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:49.800829 15461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 03:21:49.801024 15461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "5a5cf3ad69f34a808ac1302ba0c12850"
format_stamp: "Formatted at 2026-09-24 03:21:49 on dist-test-slave-00qs"
server_key: "dfcd343d33ac26da71eb510682702df6"
server_key_iv: "dd57ce6f0e8978d8c00d12019e3842d6"
server_key_version: "encryptionkey@0"
I20260924 03:21:49.801271 15461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:49.814849 15461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:49.815744 15461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:49.816090 15461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:49.835845 15461 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:34671
I20260924 03:21:49.835865 15526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:34671 every 8 connection(s)
I20260924 03:21:49.837494 15461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:21:49.841631 15527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:49.846195 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 15461
I20260924 03:21:49.846434 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:21:49.847230 10290 external_mini_cluster.cc:1428] Setting key f5e71e1719860cf05bc17b2ca85a07dc
I20260924 03:21:49.854328 15527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a5cf3ad69f34a808ac1302ba0c12850: Bootstrap starting.
I20260924 03:21:49.857506 15527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a5cf3ad69f34a808ac1302ba0c12850: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:49.858552 15527 log.cc:824] T 00000000000000000000000000000000 P 5a5cf3ad69f34a808ac1302ba0c12850: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:49.862215 15527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a5cf3ad69f34a808ac1302ba0c12850: No bootstrap required, opened a new log
I20260924 03:21:49.862344 15530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:49.849477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59528 (local address 127.10.12.190:34671)
0924 03:21:49.850568 (+ 1091us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:49.850579 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:49.850624 (+ 45us) server_negotiation.cc:407] Connection header received
0924 03:21:49.852844 (+ 2220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:49.852873 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:49.853394 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:49.853773 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:49.854395 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:49.855722 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:49.857149 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:49.857882 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:49.858060 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:49.859462 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:49.859470 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:49.859486 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:21:49.860140 (+ 654us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:49.860249 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:21:49.860277 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:49.860420 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:49.860674 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 03:21:49.861035 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":398,"threads_started":1}
I20260924 03:21:49.866585 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:21:49.871287 15527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a5cf3ad69f34a808ac1302ba0c12850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5cf3ad69f34a808ac1302ba0c12850" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 34671 } }
I20260924 03:21:49.871690 15527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a5cf3ad69f34a808ac1302ba0c12850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:49.871766 15527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a5cf3ad69f34a808ac1302ba0c12850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a5cf3ad69f34a808ac1302ba0c12850, State: Initialized, Role: FOLLOWER
I20260924 03:21:49.872388 15527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a5cf3ad69f34a808ac1302ba0c12850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5cf3ad69f34a808ac1302ba0c12850" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 34671 } }
I20260924 03:21:49.872609 15527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a5cf3ad69f34a808ac1302ba0c12850 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:21:49.872709 15527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a5cf3ad69f34a808ac1302ba0c12850 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:21:49.873000 15527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a5cf3ad69f34a808ac1302ba0c12850 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:49.875516 15527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a5cf3ad69f34a808ac1302ba0c12850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5cf3ad69f34a808ac1302ba0c12850" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 34671 } }
I20260924 03:21:49.876165 15527 leader_election.cc:302] T 00000000000000000000000000000000 P 5a5cf3ad69f34a808ac1302ba0c12850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a5cf3ad69f34a808ac1302ba0c12850; no voters:
I20260924 03:21:49.876881 15527 leader_election.cc:288] T 00000000000000000000000000000000 P 5a5cf3ad69f34a808ac1302ba0c12850 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:21:49.877017 15532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a5cf3ad69f34a808ac1302ba0c12850 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:49.877759 15532 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a5cf3ad69f34a808ac1302ba0c12850 [term 1 LEADER]: Becoming Leader. State: Replica: 5a5cf3ad69f34a808ac1302ba0c12850, State: Running, Role: LEADER
I20260924 03:21:49.878285 15532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a5cf3ad69f34a808ac1302ba0c12850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5cf3ad69f34a808ac1302ba0c12850" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 34671 } }
I20260924 03:21:49.879200 15527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a5cf3ad69f34a808ac1302ba0c12850 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:21:49.881049 15534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a5cf3ad69f34a808ac1302ba0c12850 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a5cf3ad69f34a808ac1302ba0c12850. Latest consensus state: current_term: 1 leader_uuid: "5a5cf3ad69f34a808ac1302ba0c12850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5cf3ad69f34a808ac1302ba0c12850" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 34671 } } }
I20260924 03:21:49.881587 15534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a5cf3ad69f34a808ac1302ba0c12850 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:49.881027 15533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a5cf3ad69f34a808ac1302ba0c12850 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a5cf3ad69f34a808ac1302ba0c12850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5cf3ad69f34a808ac1302ba0c12850" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 34671 } } }
I20260924 03:21:49.883483 15533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a5cf3ad69f34a808ac1302ba0c12850 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:49.883811 15537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:21:49.889389 15537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:21:49.900161 15537 catalog_manager.cc:1347] Generated new cluster ID: 4b87066603b94b5e8effbe5317a6ac9b
I20260924 03:21:49.900298 15537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:21:49.927245 15537 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:21:49.929211 15537 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:21:49.938694 15537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a5cf3ad69f34a808ac1302ba0c12850: Generated new TSK 0
I20260924 03:21:49.939872 15537 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 03:21:49.984563 15557 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 03:21:49.984753 15557 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 03:21:49.985740 15530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:49.960147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59530 (local address 127.10.12.190:34671)
0924 03:21:49.960388 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:49.960395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:49.960933 (+ 538us) server_negotiation.cc:407] Connection header received
0924 03:21:49.961288 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:49.961294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:49.961376 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:49.961520 (+ 144us) server_negotiation.cc:226] Negotiated authn=JWT
0924 03:21:49.962153 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:49.962919 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:49.965224 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:49.966107 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:49.966220 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:49.966958 (+ 738us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 03:21:49.985168 (+ 18210us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45577/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 03:21:49.985497 (+ 329us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59530: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45577/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 03:21:49.985859 15530 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59530: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45577/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 03:21:49.988061 15556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:21:49.960023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:34671 (local address 127.0.0.1:59530)
0924 03:21:49.960728 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:49.960751 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:49.960979 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:49.961577 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:49.961582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:49.961596 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0924 03:21:49.962000 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:49.962012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:49.963069 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:49.963074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:49.964915 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:49.964931 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:49.966306 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:49.966313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:49.966506 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:49.966599 (+ 93us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 03:21:49.985519 (+ 18920us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45577/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 03:21:49.985925 (+ 406us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.12.190:34671: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45577/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":327,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.10.12.190:34671: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45577/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 03:21:49.990432 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 15461
I20260924 03:21:49.991716 15522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:50.086786 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 15461
2026-09-24T03:21:50Z chronyd exiting
I20260924 03:21:50.134479 10290 test_util.cc:175] -----------------------------------------------
I20260924 03:21:50.134634 10290 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790220067268388-10290-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (776 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 03:21:50.138433 10290 mini_oidc.cc:152] Starting JWKS server
I20260924 03:21:50.140478 10290 webserver.cc:460] Webserver started at https://127.0.0.1:35915/ using document root <none> and password file <none>
I20260924 03:21:50.140727 10290 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 03:21:50.141412 10290 webserver.cc:460] Webserver started at http://127.0.0.1:42693/ using document root <none> and password file <none>
2026-09-24T03:21:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:21:50Z Disabled control of system clock
I20260924 03:21:50.168082 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:45135
--webserver_interface=127.10.12.190
--webserver_port=0
--builtin_ntp_servers=127.10.12.148:43175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:45135
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35915/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790220067268388-10290-0/testchainca.cert with env {}
W20260924 03:21:50.501513 15564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:50.501829 15564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:50.501936 15564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:50.509349 15564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:21:50.509477 15564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:50.509516 15564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:50.509564 15564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:21:50.509615 15564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 03:21:50.509703 15564 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 03:21:50.509739 15564 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35915/jwks.json
I20260924 03:21:50.520048 15564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:43175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:45135
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:45135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35915/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790220067268388-10290-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:50.521406 15564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:50.522679 15564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:50.535255 15569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:50.535355 15572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:50.535408 15570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:50.536158 15564 server_base.cc:1034] running on GCE node
I20260924 03:21:50.536983 15564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:50.538754 15564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:50.540046 15564 hybrid_clock.cc:648] HybridClock initialized: now 1790220110539997 us; error 42 us; skew 500 ppm
I20260924 03:21:50.540479 15564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:21:50.547887 15564 webserver.cc:460] Webserver started at http://127.10.12.190:46867/ using document root <none> and password file <none>
I20260924 03:21:50.548691 15564 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:50.548799 15564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:50.549125 15564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:50.552264 15564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "03359fc252e7416bba876740a0301892"
format_stamp: "Formatted at 2026-09-24 03:21:50 on dist-test-slave-00qs"
server_key: "2c3074587424853a4eab192bb6b0c2ba"
server_key_iv: "56249589fdb1a15949d862bbccd5edc5"
server_key_version: "encryptionkey@0"
I20260924 03:21:50.553133 15564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "03359fc252e7416bba876740a0301892"
format_stamp: "Formatted at 2026-09-24 03:21:50 on dist-test-slave-00qs"
server_key: "2c3074587424853a4eab192bb6b0c2ba"
server_key_iv: "56249589fdb1a15949d862bbccd5edc5"
server_key_version: "encryptionkey@0"
I20260924 03:21:50.558768 15564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:21:50.562700 15578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:50.564716 15564 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:21:50.564930 15564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "03359fc252e7416bba876740a0301892"
format_stamp: "Formatted at 2026-09-24 03:21:50 on dist-test-slave-00qs"
server_key: "2c3074587424853a4eab192bb6b0c2ba"
server_key_iv: "56249589fdb1a15949d862bbccd5edc5"
server_key_version: "encryptionkey@0"
I20260924 03:21:50.565162 15564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:50.592384 15564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:50.593374 15564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:50.593694 15564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:50.614009 15564 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:45135
I20260924 03:21:50.614027 15629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:45135 every 8 connection(s)
I20260924 03:21:50.615442 15564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:21:50.619652 15630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:50.619889 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 15564
I20260924 03:21:50.620101 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:21:50.620779 10290 external_mini_cluster.cc:1428] Setting key 061a5e725e0eaf10648133019c9ae890
I20260924 03:21:50.634169 15630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03359fc252e7416bba876740a0301892: Bootstrap starting.
I20260924 03:21:50.637044 15633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:50.622869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56444 (local address 127.10.12.190:45135)
0924 03:21:50.624544 (+ 1675us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:50.624564 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:50.624639 (+ 75us) server_negotiation.cc:407] Connection header received
0924 03:21:50.626764 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:50.626813 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:50.627653 (+ 840us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:50.628440 (+ 787us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:50.629147 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:50.630716 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:50.632153 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:50.632943 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:50.633145 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:50.634413 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:50.634425 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:50.634449 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:21:50.635052 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:50.635145 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:21:50.635170 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:50.635297 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:50.635565 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 03:21:50.635941 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":867,"thread_start_us":550,"threads_started":1}
I20260924 03:21:50.637816 15630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03359fc252e7416bba876740a0301892: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:50.639084 15630 log.cc:824] T 00000000000000000000000000000000 P 03359fc252e7416bba876740a0301892: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:50.642393 15630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03359fc252e7416bba876740a0301892: No bootstrap required, opened a new log
I20260924 03:21:50.642784 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:21:50.650892 15630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03359fc252e7416bba876740a0301892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03359fc252e7416bba876740a0301892" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45135 } }
I20260924 03:21:50.651297 15630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03359fc252e7416bba876740a0301892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:50.651364 15630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03359fc252e7416bba876740a0301892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03359fc252e7416bba876740a0301892, State: Initialized, Role: FOLLOWER
I20260924 03:21:50.652047 15630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03359fc252e7416bba876740a0301892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03359fc252e7416bba876740a0301892" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45135 } }
I20260924 03:21:50.652292 15630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03359fc252e7416bba876740a0301892 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:21:50.652410 15630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03359fc252e7416bba876740a0301892 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:21:50.652537 15630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03359fc252e7416bba876740a0301892 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:50.654780 15630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03359fc252e7416bba876740a0301892 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03359fc252e7416bba876740a0301892" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45135 } }
I20260924 03:21:50.655297 15630 leader_election.cc:302] T 00000000000000000000000000000000 P 03359fc252e7416bba876740a0301892 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03359fc252e7416bba876740a0301892; no voters:
I20260924 03:21:50.655988 15630 leader_election.cc:288] T 00000000000000000000000000000000 P 03359fc252e7416bba876740a0301892 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:21:50.656162 15635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03359fc252e7416bba876740a0301892 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:50.656816 15635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03359fc252e7416bba876740a0301892 [term 1 LEADER]: Becoming Leader. State: Replica: 03359fc252e7416bba876740a0301892, State: Running, Role: LEADER
I20260924 03:21:50.657397 15635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03359fc252e7416bba876740a0301892 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03359fc252e7416bba876740a0301892" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45135 } }
I20260924 03:21:50.658105 15630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03359fc252e7416bba876740a0301892 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:21:50.660207 15637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03359fc252e7416bba876740a0301892 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03359fc252e7416bba876740a0301892. Latest consensus state: current_term: 1 leader_uuid: "03359fc252e7416bba876740a0301892" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03359fc252e7416bba876740a0301892" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45135 } } }
I20260924 03:21:50.660738 15637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03359fc252e7416bba876740a0301892 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:50.660334 15636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03359fc252e7416bba876740a0301892 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03359fc252e7416bba876740a0301892" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03359fc252e7416bba876740a0301892" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45135 } } }
I20260924 03:21:50.662556 15636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03359fc252e7416bba876740a0301892 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:50.664119 15644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:21:50.669361 15644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:21:50.679147 15644 catalog_manager.cc:1347] Generated new cluster ID: 2b4036b9cf2340a3a638142c819fddb3
I20260924 03:21:50.679286 15644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:21:50.695905 15644 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:21:50.697891 15644 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:21:50.707069 15644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03359fc252e7416bba876740a0301892: Generated new TSK 0
I20260924 03:21:50.708243 15644 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 03:21:50.752377 15660 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 03:21:50.752523 15660 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 03:21:50.753509 15633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:50.724765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56450 (local address 127.10.12.190:45135)
0924 03:21:50.725063 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:50.725070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:50.725350 (+ 280us) server_negotiation.cc:407] Connection header received
0924 03:21:50.725628 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:50.725632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:50.725715 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:50.725858 (+ 143us) server_negotiation.cc:226] Negotiated authn=JWT
0924 03:21:50.726376 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:50.727162 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:50.729335 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:50.729844 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:50.729973 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:50.730669 (+ 696us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 03:21:50.752934 (+ 22265us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35915/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 03:21:50.753263 (+ 329us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56450: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35915/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":95}
W20260924 03:21:50.753629 15633 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56450: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35915/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 03:21:50.753903 15659 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:21:50.724563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:45135 (local address 127.0.0.1:56450)
0924 03:21:50.725229 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:50.725251 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:50.725421 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:50.725902 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:50.725908 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:50.725921 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0924 03:21:50.726205 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:50.726216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:50.727330 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:50.727336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:50.729120 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:50.729138 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:50.730061 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:50.730068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:50.730293 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:50.730348 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 03:21:50.753199 (+ 22851us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35915/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 03:21:50.753504 (+ 305us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.12.190:45135: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35915/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":355,"threads_started":1}
I20260924 03:21:50.755100 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 15564
I20260924 03:21:50.756392 15625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:50.852227 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 15564
2026-09-24T03:21:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (766 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 03:21:50.904986 10290 mini_oidc.cc:152] Starting JWKS server
I20260924 03:21:50.906896 10290 webserver.cc:460] Webserver started at https://127.0.0.1:33139/ using document root <none> and password file <none>
I20260924 03:21:50.907160 10290 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 03:21:50.907874 10290 webserver.cc:460] Webserver started at http://127.0.0.1:46107/ using document root <none> and password file <none>
2026-09-24T03:21:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:21:50Z Disabled control of system clock
I20260924 03:21:50.934333 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:46649
--webserver_interface=127.10.12.190
--webserver_port=0
--builtin_ntp_servers=127.10.12.148:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:46649
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33139/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 03:21:51.270829 15667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:51.271211 15667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:51.271338 15667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:51.279699 15667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:21:51.279876 15667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:51.279996 15667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:51.280087 15667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:21:51.280154 15667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 03:21:51.280246 15667 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 03:21:51.280303 15667 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33139/jwks.json
I20260924 03:21:51.290234 15667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:46649
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:46649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33139/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:51.291656 15667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:51.293663 15667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:51.307191 15672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:51.307171 15675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:51.307469 15673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:51.308233 15667 server_base.cc:1034] running on GCE node
I20260924 03:21:51.309057 15667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:51.310647 15667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:51.311877 15667 hybrid_clock.cc:648] HybridClock initialized: now 1790220111311817 us; error 54 us; skew 500 ppm
I20260924 03:21:51.312283 15667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:21:51.314754 15667 webserver.cc:460] Webserver started at http://127.10.12.190:41741/ using document root <none> and password file <none>
I20260924 03:21:51.315491 15667 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:51.315583 15667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:51.315862 15667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:51.318990 15667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "91f9eea3eb9b4df1b6841a64f6066149"
format_stamp: "Formatted at 2026-09-24 03:21:51 on dist-test-slave-00qs"
server_key: "69a8ca2fe5a35bbb5e53f5beb47242da"
server_key_iv: "9d54efc514f46f5509913da9dfb7c9c3"
server_key_version: "encryptionkey@0"
I20260924 03:21:51.319904 15667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "91f9eea3eb9b4df1b6841a64f6066149"
format_stamp: "Formatted at 2026-09-24 03:21:51 on dist-test-slave-00qs"
server_key: "69a8ca2fe5a35bbb5e53f5beb47242da"
server_key_iv: "9d54efc514f46f5509913da9dfb7c9c3"
server_key_version: "encryptionkey@0"
I20260924 03:21:51.326200 15667 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260924 03:21:51.330001 15681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:51.331686 15667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:21:51.331845 15667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "91f9eea3eb9b4df1b6841a64f6066149"
format_stamp: "Formatted at 2026-09-24 03:21:51 on dist-test-slave-00qs"
server_key: "69a8ca2fe5a35bbb5e53f5beb47242da"
server_key_iv: "9d54efc514f46f5509913da9dfb7c9c3"
server_key_version: "encryptionkey@0"
I20260924 03:21:51.332119 15667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:51.348008 15667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:51.348838 15667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:51.349115 15667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:51.367741 15667 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:46649
I20260924 03:21:51.367805 15732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:46649 every 8 connection(s)
I20260924 03:21:51.369131 15667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:21:51.373044 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:51.375655 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 15667
I20260924 03:21:51.375877 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:21:51.376668 10290 external_mini_cluster.cc:1428] Setting key 4382e005cf8971917479df949e5868f0
I20260924 03:21:51.385368 15733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f9eea3eb9b4df1b6841a64f6066149: Bootstrap starting.
I20260924 03:21:51.388809 15733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91f9eea3eb9b4df1b6841a64f6066149: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:51.390007 15733 log.cc:824] T 00000000000000000000000000000000 P 91f9eea3eb9b4df1b6841a64f6066149: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:51.391098 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:51.378767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56542 (local address 127.10.12.190:46649)
0924 03:21:51.379900 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:51.379915 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:51.379995 (+ 80us) server_negotiation.cc:407] Connection header received
0924 03:21:51.381744 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:51.381770 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:51.382326 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:51.382747 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:51.383349 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:51.384709 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:51.386095 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:51.386801 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:51.387030 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:51.388289 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:51.388297 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:51.388313 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:21:51.388967 (+ 654us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:51.389075 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:21:51.389100 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:51.389223 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:51.389467 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 03:21:51.389805 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":365,"threads_started":1}
I20260924 03:21:51.393352 15733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f9eea3eb9b4df1b6841a64f6066149: No bootstrap required, opened a new log
I20260924 03:21:51.395615 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:21:51.402104 15733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91f9eea3eb9b4df1b6841a64f6066149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f9eea3eb9b4df1b6841a64f6066149" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46649 } }
I20260924 03:21:51.402522 15733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91f9eea3eb9b4df1b6841a64f6066149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:51.402619 15733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91f9eea3eb9b4df1b6841a64f6066149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f9eea3eb9b4df1b6841a64f6066149, State: Initialized, Role: FOLLOWER
I20260924 03:21:51.403205 15733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91f9eea3eb9b4df1b6841a64f6066149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f9eea3eb9b4df1b6841a64f6066149" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46649 } }
I20260924 03:21:51.403420 15733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91f9eea3eb9b4df1b6841a64f6066149 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:21:51.403558 15733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91f9eea3eb9b4df1b6841a64f6066149 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:21:51.403721 15733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91f9eea3eb9b4df1b6841a64f6066149 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:51.406563 15733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91f9eea3eb9b4df1b6841a64f6066149 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f9eea3eb9b4df1b6841a64f6066149" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46649 } }
I20260924 03:21:51.407130 15733 leader_election.cc:302] T 00000000000000000000000000000000 P 91f9eea3eb9b4df1b6841a64f6066149 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91f9eea3eb9b4df1b6841a64f6066149; no voters:
I20260924 03:21:51.407753 15733 leader_election.cc:288] T 00000000000000000000000000000000 P 91f9eea3eb9b4df1b6841a64f6066149 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:21:51.407985 15738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91f9eea3eb9b4df1b6841a64f6066149 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:51.409039 15738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91f9eea3eb9b4df1b6841a64f6066149 [term 1 LEADER]: Becoming Leader. State: Replica: 91f9eea3eb9b4df1b6841a64f6066149, State: Running, Role: LEADER
I20260924 03:21:51.409627 15738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91f9eea3eb9b4df1b6841a64f6066149 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f9eea3eb9b4df1b6841a64f6066149" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46649 } }
I20260924 03:21:51.410020 15733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91f9eea3eb9b4df1b6841a64f6066149 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:21:51.412396 15740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f9eea3eb9b4df1b6841a64f6066149 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91f9eea3eb9b4df1b6841a64f6066149. Latest consensus state: current_term: 1 leader_uuid: "91f9eea3eb9b4df1b6841a64f6066149" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f9eea3eb9b4df1b6841a64f6066149" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46649 } } }
I20260924 03:21:51.412817 15740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f9eea3eb9b4df1b6841a64f6066149 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:51.413488 15739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f9eea3eb9b4df1b6841a64f6066149 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91f9eea3eb9b4df1b6841a64f6066149" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f9eea3eb9b4df1b6841a64f6066149" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46649 } } }
I20260924 03:21:51.413827 15739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f9eea3eb9b4df1b6841a64f6066149 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:51.416652 15747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:21:51.422652 15747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:21:51.432679 15747 catalog_manager.cc:1347] Generated new cluster ID: 93a69aed5d19479ba6d4d027eeb0cae7
I20260924 03:21:51.432818 15747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:21:51.457185 15747 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:21:51.458544 15747 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:21:51.473541 15747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91f9eea3eb9b4df1b6841a64f6066149: Generated new TSK 0
I20260924 03:21:51.474455 15747 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:21:51.497042 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:51.488326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56554 (local address 127.10.12.190:46649)
0924 03:21:51.488538 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:51.488545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:51.488809 (+ 264us) server_negotiation.cc:407] Connection header received
0924 03:21:51.489061 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:51.489068 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:51.489171 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:51.489340 (+ 169us) server_negotiation.cc:226] Negotiated authn=JWT
0924 03:21:51.489778 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:51.490944 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:51.492473 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:51.493177 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:51.493315 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:51.493930 (+ 615us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 03:21:51.496360 (+ 2430us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33139/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 03:21:51.496729 (+ 369us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56554: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33139/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 03:21:51.497177 15736 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56554: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33139/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 03:21:51.498934 15762 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:21:51.488146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:46649 (local address 127.0.0.1:56554)
0924 03:21:51.488685 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:51.488704 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:51.488864 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:51.489364 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:51.489368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:51.489377 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 03:21:51.489634 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:51.489642 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:51.491122 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:51.491126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:51.492306 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:51.492318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:51.493441 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:51.493446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:51.493594 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:51.493631 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 03:21:51.496595 (+ 2964us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33139/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 03:21:51.498477 (+ 1882us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.12.190:46649: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33139/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":1540,"thread_start_us":407,"threads_started":1}
I20260924 03:21:51.500351 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 15667
I20260924 03:21:51.501847 15728 generic_service.cc:196] Checking for leaks (request via RPC)
E20260924 03:21:51.511376 15763 webserver.cc:533] Webserver: sslize failed:
E20260924 03:21:51.511536 15763 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 03:21:51.598292 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 15667
2026-09-24T03:21:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (745 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T03:21:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:21:51Z Disabled control of system clock
I20260924 03:21:51.684422 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:46535
--webserver_interface=127.10.12.190
--webserver_port=0
--builtin_ntp_servers=127.10.12.148:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:46535
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790220067268388-10290-0/testchainca.cert with env {}
W20260924 03:21:52.033867 15768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:52.034422 15768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:52.034576 15768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:52.042191 15768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:21:52.042351 15768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:52.042418 15768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:52.042501 15768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:21:52.042567 15768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 03:21:52.042659 15768 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 03:21:52.042712 15768 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 03:21:52.055567 15768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:46535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:46535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790220067268388-10290-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:52.057090 15768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:52.058529 15768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:52.070956 15773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:52.071009 15774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:52.071030 15776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:52.071350 15768 server_base.cc:1034] running on GCE node
I20260924 03:21:52.072494 15768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:52.074059 15768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:52.075348 15768 hybrid_clock.cc:648] HybridClock initialized: now 1790220112075286 us; error 38 us; skew 500 ppm
I20260924 03:21:52.075749 15768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:21:52.078451 15768 webserver.cc:460] Webserver started at http://127.10.12.190:43791/ using document root <none> and password file <none>
I20260924 03:21:52.079224 15768 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:52.079361 15768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:52.079677 15768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:52.082993 15768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "8d0b170523a249ca8bcfc1a1e536f0f8"
format_stamp: "Formatted at 2026-09-24 03:21:52 on dist-test-slave-00qs"
server_key: "eb3d5f7d9694d3961631d5d727138d20"
server_key_iv: "6beb3d917c35c03f6d90515f6b3d521a"
server_key_version: "encryptionkey@0"
I20260924 03:21:52.083837 15768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "8d0b170523a249ca8bcfc1a1e536f0f8"
format_stamp: "Formatted at 2026-09-24 03:21:52 on dist-test-slave-00qs"
server_key: "eb3d5f7d9694d3961631d5d727138d20"
server_key_iv: "6beb3d917c35c03f6d90515f6b3d521a"
server_key_version: "encryptionkey@0"
I20260924 03:21:52.089509 15768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 03:21:52.093757 15782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:52.095286 15768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 03:21:52.095494 15768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "8d0b170523a249ca8bcfc1a1e536f0f8"
format_stamp: "Formatted at 2026-09-24 03:21:52 on dist-test-slave-00qs"
server_key: "eb3d5f7d9694d3961631d5d727138d20"
server_key_iv: "6beb3d917c35c03f6d90515f6b3d521a"
server_key_version: "encryptionkey@0"
I20260924 03:21:52.095721 15768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:52.118376 15768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:52.119417 15768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:52.119987 15768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:52.140465 15768 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:46535
I20260924 03:21:52.140501 15833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:46535 every 8 connection(s)
I20260924 03:21:52.141984 15768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:21:52.143412 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 15768
I20260924 03:21:52.143922 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:21:52.144685 10290 external_mini_cluster.cc:1428] Setting key c1177557bcbef9bc3c1bfffd0d39a70a
I20260924 03:21:52.147001 15834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:52.160446 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:52.146762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35464 (local address 127.10.12.190:46535)
0924 03:21:52.148357 (+ 1595us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:52.148375 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:52.148442 (+ 67us) server_negotiation.cc:407] Connection header received
0924 03:21:52.150728 (+ 2286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:52.150841 (+ 113us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:52.151530 (+ 689us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:52.152037 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:52.152720 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:52.154150 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:52.155525 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:52.156337 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:52.156532 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:52.157867 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:52.157879 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:52.157896 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:21:52.158532 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:52.158630 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:21:52.158654 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:52.158769 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:52.159029 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 03:21:52.159373 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":872,"thread_start_us":528,"threads_started":1}
I20260924 03:21:52.160807 15834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d0b170523a249ca8bcfc1a1e536f0f8: Bootstrap starting.
I20260924 03:21:52.164175 15834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d0b170523a249ca8bcfc1a1e536f0f8: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:52.165122 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:21:52.165395 15834 log.cc:824] T 00000000000000000000000000000000 P 8d0b170523a249ca8bcfc1a1e536f0f8: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:52.168535 15834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d0b170523a249ca8bcfc1a1e536f0f8: No bootstrap required, opened a new log
I20260924 03:21:52.176064 15834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d0b170523a249ca8bcfc1a1e536f0f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0b170523a249ca8bcfc1a1e536f0f8" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46535 } }
I20260924 03:21:52.176522 15834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d0b170523a249ca8bcfc1a1e536f0f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:52.176617 15834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d0b170523a249ca8bcfc1a1e536f0f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d0b170523a249ca8bcfc1a1e536f0f8, State: Initialized, Role: FOLLOWER
I20260924 03:21:52.177306 15834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d0b170523a249ca8bcfc1a1e536f0f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0b170523a249ca8bcfc1a1e536f0f8" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46535 } }
I20260924 03:21:52.177574 15834 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d0b170523a249ca8bcfc1a1e536f0f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:21:52.177707 15834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d0b170523a249ca8bcfc1a1e536f0f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:21:52.177871 15834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d0b170523a249ca8bcfc1a1e536f0f8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:52.180266 15834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d0b170523a249ca8bcfc1a1e536f0f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0b170523a249ca8bcfc1a1e536f0f8" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46535 } }
I20260924 03:21:52.180850 15834 leader_election.cc:302] T 00000000000000000000000000000000 P 8d0b170523a249ca8bcfc1a1e536f0f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d0b170523a249ca8bcfc1a1e536f0f8; no voters:
I20260924 03:21:52.181676 15834 leader_election.cc:288] T 00000000000000000000000000000000 P 8d0b170523a249ca8bcfc1a1e536f0f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:21:52.182180 15839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d0b170523a249ca8bcfc1a1e536f0f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:52.183234 15839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d0b170523a249ca8bcfc1a1e536f0f8 [term 1 LEADER]: Becoming Leader. State: Replica: 8d0b170523a249ca8bcfc1a1e536f0f8, State: Running, Role: LEADER
I20260924 03:21:52.184055 15834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d0b170523a249ca8bcfc1a1e536f0f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:21:52.184098 15839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d0b170523a249ca8bcfc1a1e536f0f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0b170523a249ca8bcfc1a1e536f0f8" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46535 } }
I20260924 03:21:52.188055 15840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d0b170523a249ca8bcfc1a1e536f0f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d0b170523a249ca8bcfc1a1e536f0f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0b170523a249ca8bcfc1a1e536f0f8" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46535 } } }
I20260924 03:21:52.188550 15840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d0b170523a249ca8bcfc1a1e536f0f8 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:52.188237 15841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d0b170523a249ca8bcfc1a1e536f0f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d0b170523a249ca8bcfc1a1e536f0f8. Latest consensus state: current_term: 1 leader_uuid: "8d0b170523a249ca8bcfc1a1e536f0f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0b170523a249ca8bcfc1a1e536f0f8" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46535 } } }
I20260924 03:21:52.188985 15841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d0b170523a249ca8bcfc1a1e536f0f8 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:52.190382 15849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:21:52.200103 15849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:21:52.211323 15849 catalog_manager.cc:1347] Generated new cluster ID: 8f20c9c816be4d12b888ea005425163e
I20260924 03:21:52.211483 15849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:21:52.225281 15849 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:21:52.227265 15849 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:21:52.238555 15849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d0b170523a249ca8bcfc1a1e536f0f8: Generated new TSK 0
I20260924 03:21:52.239868 15849 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:21:53.221661 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:52.270040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35480 (local address 127.10.12.190:46535)
0924 03:21:52.270348 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:52.270357 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:52.270944 (+ 587us) server_negotiation.cc:407] Connection header received
0924 03:21:52.271254 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:52.271259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:52.271353 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:52.271507 (+ 154us) server_negotiation.cc:226] Negotiated authn=JWT
0924 03:21:52.272081 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:52.272889 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:52.275216 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:52.275887 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:52.276065 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:52.276738 (+ 673us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 03:21:53.220938 (+944200us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 03:21:53.221382 (+ 444us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35480: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":96}
W20260924 03:21:53.221807 15837 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35480: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 03:21:53.222151 15863 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:21:52.269902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:46535 (local address 127.0.0.1:35480)
0924 03:21:52.270728 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:52.270771 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:52.270981 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:52.271542 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:52.271547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:52.271560 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0924 03:21:52.271883 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:52.271895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:52.273038 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:52.273044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:52.274992 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:52.275014 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:52.276141 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:52.276149 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:52.276354 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:52.276408 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 03:21:53.221369 (+944961us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 03:21:53.221732 (+ 363us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.12.190:46535: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":417,"threads_started":1}
I20260924 03:21:53.223628 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 15768
I20260924 03:21:53.225221 15829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:53.344399 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 15768
2026-09-24T03:21:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1731 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (1506 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (448 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:21:55 dist-test-slave-00qs krb5kdc[15879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:21:55 dist-test-slave-00qs krb5kdc[15879](info): set up 2 sockets
Sep 24 03:21:55 dist-test-slave-00qs krb5kdc[15879](info): commencing operation
krb5kdc: starting...
W20260924 03:21:56.332760 10290 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:21:56 dist-test-slave-00qs krb5kdc[15879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:21:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:21:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:21:56.548913 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:38241
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:36797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:38241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:56.882939 15892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:56.883258 15892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:56.883381 15892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:56.891052 15892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:21:56.891183 15892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:56.891233 15892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:56.891312 15892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:21:56.891366 15892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:21:56.901757 15892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:36797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:38241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:38241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:56.903353 15892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:56.904798 15892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:56.917637 15898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:56.917797 15900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:56.917810 15897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:56.918318 15892 server_base.cc:1034] running on GCE node
I20260924 03:21:56.919358 15892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:56.920939 15892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:56.922192 15892 hybrid_clock.cc:648] HybridClock initialized: now 1790220116922142 us; error 36 us; skew 500 ppm
I20260924 03:21:56.922580 15892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:56 dist-test-slave-00qs krb5kdc[15879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220116, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:56.931378 15892 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:21:56.933182 15892 webserver.cc:460] Webserver started at http://127.10.12.190:42449/ using document root <none> and password file <none>
I20260924 03:21:56.933921 15892 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:56.934044 15892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:56.934384 15892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:56.937675 15892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "899685aa52e14b639a512198f032c654"
format_stamp: "Formatted at 2026-09-24 03:21:56 on dist-test-slave-00qs"
server_key: "fb1d753731cc405bfb0a04899756a37b"
server_key_iv: "16082c0e5c105296887a1d4b693449a0"
server_key_version: "encryptionkey@0"
I20260924 03:21:56.938505 15892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "899685aa52e14b639a512198f032c654"
format_stamp: "Formatted at 2026-09-24 03:21:56 on dist-test-slave-00qs"
server_key: "fb1d753731cc405bfb0a04899756a37b"
server_key_iv: "16082c0e5c105296887a1d4b693449a0"
server_key_version: "encryptionkey@0"
I20260924 03:21:56.944026 15892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 03:21:56.947871 15907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:56.949633 15892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 03:21:56.949808 15892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "899685aa52e14b639a512198f032c654"
format_stamp: "Formatted at 2026-09-24 03:21:56 on dist-test-slave-00qs"
server_key: "fb1d753731cc405bfb0a04899756a37b"
server_key_iv: "16082c0e5c105296887a1d4b693449a0"
server_key_version: "encryptionkey@0"
I20260924 03:21:56.950029 15892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:56.970587 15892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:56.975365 15892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:56.975651 15892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:56.995908 15892 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:38241
I20260924 03:21:56.995929 15958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:38241 every 8 connection(s)
I20260924 03:21:56.997381 15892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:21:57.000492 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 15892
I20260924 03:21:57.000689 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:21:57.001319 10290 external_mini_cluster.cc:1428] Setting key d1375f1d1be66a71d1202ea3bd7c8951
I20260924 03:21:57.001590 15959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:57.016788 15959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 899685aa52e14b639a512198f032c654: Bootstrap starting.
Sep 24 03:21:57 dist-test-slave-00qs krb5kdc[15879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:57.020627 15959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 899685aa52e14b639a512198f032c654: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:57.022055 15959 log.cc:824] T 00000000000000000000000000000000 P 899685aa52e14b639a512198f032c654: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:57.025797 15959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 899685aa52e14b639a512198f032c654: No bootstrap required, opened a new log
I20260924 03:21:57.027870 15962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:57.003283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39304 (local address 127.10.12.190:38241)
0924 03:21:57.004125 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:57.004140 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:57.004191 (+ 51us) server_negotiation.cc:407] Connection header received
0924 03:21:57.006678 (+ 2487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:57.006715 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:57.007349 (+ 634us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:57.007866 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:57.009266 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:57.011266 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:57.013342 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:57.014513 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:57.014728 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:57.019018 (+ 4290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:57.019030 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:57.019050 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:57.019106 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:57.023139 (+ 4033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:57.024076 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:57.024085 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:57.024094 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:57.024350 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:57.024795 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:57.024800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:57.024803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:57.025534 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:57.025830 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:57.026351 (+ 521us) server_negotiation.cc:299] Negotiation successful
0924 03:21:57.026785 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":329,"threads_started":1}
I20260924 03:21:57.034626 15959 raft_consensus.cc:348] T 00000000000000000000000000000000 P 899685aa52e14b639a512198f032c654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899685aa52e14b639a512198f032c654" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 38241 } }
I20260924 03:21:57.035029 15959 raft_consensus.cc:374] T 00000000000000000000000000000000 P 899685aa52e14b639a512198f032c654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:21:57.035126 15959 raft_consensus.cc:729] T 00000000000000000000000000000000 P 899685aa52e14b639a512198f032c654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 899685aa52e14b639a512198f032c654, State: Initialized, Role: FOLLOWER
I20260924 03:21:57.035809 15959 consensus_queue.cc:260] T 00000000000000000000000000000000 P 899685aa52e14b639a512198f032c654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899685aa52e14b639a512198f032c654" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 38241 } }
I20260924 03:21:57.036092 15959 raft_consensus.cc:388] T 00000000000000000000000000000000 P 899685aa52e14b639a512198f032c654 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:21:57.036233 15959 raft_consensus.cc:482] T 00000000000000000000000000000000 P 899685aa52e14b639a512198f032c654 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:21:57.036381 15959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 899685aa52e14b639a512198f032c654 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:57.038724 15959 raft_consensus.cc:504] T 00000000000000000000000000000000 P 899685aa52e14b639a512198f032c654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899685aa52e14b639a512198f032c654" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 38241 } }
I20260924 03:21:57.039300 15959 leader_election.cc:302] T 00000000000000000000000000000000 P 899685aa52e14b639a512198f032c654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 899685aa52e14b639a512198f032c654; no voters:
I20260924 03:21:57.040009 15959 leader_election.cc:288] T 00000000000000000000000000000000 P 899685aa52e14b639a512198f032c654 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:21:57.040375 15964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 899685aa52e14b639a512198f032c654 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:57.041208 15964 raft_consensus.cc:686] T 00000000000000000000000000000000 P 899685aa52e14b639a512198f032c654 [term 1 LEADER]: Becoming Leader. State: Replica: 899685aa52e14b639a512198f032c654, State: Running, Role: LEADER
I20260924 03:21:57.041759 15964 consensus_queue.cc:237] T 00000000000000000000000000000000 P 899685aa52e14b639a512198f032c654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899685aa52e14b639a512198f032c654" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 38241 } }
I20260924 03:21:57.042332 15959 sys_catalog.cc:564] T 00000000000000000000000000000000 P 899685aa52e14b639a512198f032c654 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:21:57.044567 15965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899685aa52e14b639a512198f032c654 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "899685aa52e14b639a512198f032c654" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899685aa52e14b639a512198f032c654" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 38241 } } }
I20260924 03:21:57.045074 15965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899685aa52e14b639a512198f032c654 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:57.045231 15966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899685aa52e14b639a512198f032c654 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 899685aa52e14b639a512198f032c654. Latest consensus state: current_term: 1 leader_uuid: "899685aa52e14b639a512198f032c654" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899685aa52e14b639a512198f032c654" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 38241 } } }
I20260924 03:21:57.045642 15966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899685aa52e14b639a512198f032c654 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:57.048333 15970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:21:57.053962 15970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:21:57.055100 15981 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 03:21:57.064275 15970 catalog_manager.cc:1347] Generated new cluster ID: 2bb9580e31aa4c09b1b8627207d65b48
I20260924 03:21:57.064404 15970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:21:57.085458 15970 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:21:57.086946 15970 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:21:57.094673 15970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 899685aa52e14b639a512198f032c654: Generated new TSK 0
I20260924 03:21:57.095574 15970 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
I20260924 03:21:57.152915 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:38241
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:36797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:57.505620 15985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:57.505927 15985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:57.506052 15985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:57.513922 15985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:57.514047 15985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:57.514201 15985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:21:57.524547 15985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:36797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:38241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:57.525888 15985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:57.527459 15985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:57.541111 15993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:57.541141 15990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:57.541337 15991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:57.543035 15985 server_base.cc:1034] running on GCE node
I20260924 03:21:57.543885 15985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:57.545046 15985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:57.546301 15985 hybrid_clock.cc:648] HybridClock initialized: now 1790220117546241 us; error 52 us; skew 500 ppm
I20260924 03:21:57.546669 15985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:57 dist-test-slave-00qs krb5kdc[15879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220117, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:57.550082 15985 init.cc:377] Logged in from keytab as kudu/127.10.12.129@KRBTEST.COM (short username kudu)
I20260924 03:21:57.551761 15985 webserver.cc:460] Webserver started at http://127.10.12.129:38145/ using document root <none> and password file <none>
I20260924 03:21:57.552639 15985 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:57.552773 15985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:57.553093 15985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:57.556262 15985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "f4ff37d599ed4ccd988f3d2fbd68c601"
format_stamp: "Formatted at 2026-09-24 03:21:57 on dist-test-slave-00qs"
server_key: "18c9536690c2d734052500429feecccf"
server_key_iv: "c1463c142a4c2938b2ec66794af5f615"
server_key_version: "encryptionkey@0"
I20260924 03:21:57.557103 15985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "f4ff37d599ed4ccd988f3d2fbd68c601"
format_stamp: "Formatted at 2026-09-24 03:21:57 on dist-test-slave-00qs"
server_key: "18c9536690c2d734052500429feecccf"
server_key_iv: "c1463c142a4c2938b2ec66794af5f615"
server_key_version: "encryptionkey@0"
I20260924 03:21:57.562671 15985 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 03:21:57.566422 16000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:57.568037 15985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 03:21:57.568213 15985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "f4ff37d599ed4ccd988f3d2fbd68c601"
format_stamp: "Formatted at 2026-09-24 03:21:57 on dist-test-slave-00qs"
server_key: "18c9536690c2d734052500429feecccf"
server_key_iv: "c1463c142a4c2938b2ec66794af5f615"
server_key_version: "encryptionkey@0"
I20260924 03:21:57.568426 15985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:57.591140 15985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:57.596093 15985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:57.596382 15985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:57.597568 15985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:21:57.599030 15985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:57.599123 15985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:57.599327 15985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:57.599413 15985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:57.632558 15985 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:43323
I20260924 03:21:57.632622 16112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:43323 every 8 connection(s)
I20260924 03:21:57.633894 15985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:21:57.637712 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 15985
I20260924 03:21:57.637974 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:21:57.638772 10290 external_mini_cluster.cc:1428] Setting key 32e3794cbae8fd1e2f0f2a68b5c4e6e5
Sep 24 03:21:57 dist-test-slave-00qs krb5kdc[15879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220117, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:57.658967 16115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:57.636486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:52425 (local address 127.10.12.190:38241)
0924 03:21:57.636987 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:57.636994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:57.638183 (+ 1189us) server_negotiation.cc:407] Connection header received
0924 03:21:57.640709 (+ 2526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:57.640713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:57.640785 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:57.640907 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:57.643183 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:57.644101 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:57.646627 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:57.647493 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:57.647826 (+ 333us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:57.651656 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:57.651664 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:57.651668 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:57.651714 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:57.655172 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:57.656149 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:57.656155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:57.656158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:57.656351 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:57.656850 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:57.656855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:57.656859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:57.657263 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:57.657453 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:57.658298 (+ 845us) server_negotiation.cc:299] Negotiation successful
0924 03:21:57.658579 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":252,"threads_started":1}
I20260924 03:21:57.659865 16116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:57.637116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:38241 (local address 127.10.12.129:52425)
0924 03:21:57.637992 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:57.638055 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:57.640282 (+ 2227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:57.641203 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:57.641216 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:57.642015 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:57.642947 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:57.642976 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:57.644302 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:57.644307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:57.646223 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:57.646243 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:57.647894 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:57.647903 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:57.648257 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:57.648977 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:57.649002 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:57.651288 (+ 2286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:57.655405 (+ 4117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:57.655415 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:57.655431 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:57.656016 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:57.656501 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:57.656505 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:57.656508 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:57.656717 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:57.657452 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:57.657460 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:57.657862 (+ 402us) client_negotiation.cc:769] Sending connection context
0924 03:21:57.658122 (+ 260us) client_negotiation.cc:240] Negotiation successful
0924 03:21:57.658567 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":343,"threads_started":1}
WARNING: no policy specified for kudu/127.10.12.130@KRBTEST.COM; defaulting to no policy
I20260924 03:21:57.661630 16113 heartbeater.cc:347] Connected to a master server at 127.10.12.190:38241
I20260924 03:21:57.662012 16113 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:57.662762 16113 heartbeater.cc:511] Master 127.10.12.190:38241 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.130@KRBTEST.COM" created.
I20260924 03:21:57.665695 15924 ts_manager.cc:194] Registered new tserver with Master: f4ff37d599ed4ccd988f3d2fbd68c601 (127.10.12.129:43323)
I20260924 03:21:57.667702 15924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.129@KRBTEST.COM'} at 127.10.12.129:52425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
I20260924 03:21:57.693248 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:38241
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:36797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:58.035780 16119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:58.036141 16119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:58.036288 16119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:58.043882 16119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:58.044056 16119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:58.044207 16119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:21:58.055694 16119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:36797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:38241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:58.057130 16119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:58.058451 16119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:58.072692 16125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:58.072961 16127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:58.073356 16124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:58.073700 16119 server_base.cc:1034] running on GCE node
I20260924 03:21:58.074323 16119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:58.075407 16119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:58.076653 16119 hybrid_clock.cc:648] HybridClock initialized: now 1790220118076586 us; error 60 us; skew 500 ppm
I20260924 03:21:58.077008 16119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:58 dist-test-slave-00qs krb5kdc[15879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220118, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:58.080827 16119 init.cc:377] Logged in from keytab as kudu/127.10.12.130@KRBTEST.COM (short username kudu)
I20260924 03:21:58.082510 16119 webserver.cc:460] Webserver started at http://127.10.12.130:34135/ using document root <none> and password file <none>
I20260924 03:21:58.083227 16119 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:58.083339 16119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:58.083652 16119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:58.086730 16119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "d4ef28a16be945d7a863f99495896aab"
format_stamp: "Formatted at 2026-09-24 03:21:58 on dist-test-slave-00qs"
server_key: "844b890965a3df03fbcc01e0db357f36"
server_key_iv: "27a4a46947b21fd9ded5549bb6a32a10"
server_key_version: "encryptionkey@0"
I20260924 03:21:58.087556 16119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "d4ef28a16be945d7a863f99495896aab"
format_stamp: "Formatted at 2026-09-24 03:21:58 on dist-test-slave-00qs"
server_key: "844b890965a3df03fbcc01e0db357f36"
server_key_iv: "27a4a46947b21fd9ded5549bb6a32a10"
server_key_version: "encryptionkey@0"
I20260924 03:21:58.093269 16119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 03:21:58.097430 16134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:58.099171 16119 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260924 03:21:58.099377 16119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "d4ef28a16be945d7a863f99495896aab"
format_stamp: "Formatted at 2026-09-24 03:21:58 on dist-test-slave-00qs"
server_key: "844b890965a3df03fbcc01e0db357f36"
server_key_iv: "27a4a46947b21fd9ded5549bb6a32a10"
server_key_version: "encryptionkey@0"
I20260924 03:21:58.099614 16119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:58.121985 16119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:58.126463 16119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:58.126751 16119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:58.127799 16119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:21:58.129294 16119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:58.129382 16119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:58.129505 16119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:58.129557 16119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:58.163520 16119 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:41093
I20260924 03:21:58.163658 16246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:41093 every 8 connection(s)
I20260924 03:21:58.165010 16119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:21:58.165289 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 16119
I20260924 03:21:58.165465 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:21:58.166064 10290 external_mini_cluster.cc:1428] Setting key ae61a3234f89f529d1e62bcaf11f551c
Sep 24 03:21:58 dist-test-slave-00qs krb5kdc[15879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220118, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.12.131@KRBTEST.COM; defaulting to no policy
I20260924 03:21:58.187711 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:58.167838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:37141 (local address 127.10.12.190:38241)
0924 03:21:58.168344 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:58.168350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:58.169540 (+ 1190us) server_negotiation.cc:407] Connection header received
0924 03:21:58.172215 (+ 2675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:58.172219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:58.172310 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:58.172441 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:58.174578 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:58.175372 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:58.177091 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:58.177545 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:58.177662 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:58.181540 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:58.181546 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:58.181549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:58.181587 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:58.184285 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:58.185170 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:58.185174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:58.185177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:58.185316 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:58.185786 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:58.185789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:58.185791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:58.186110 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:58.186249 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:58.187049 (+ 800us) server_negotiation.cc:299] Negotiation successful
0924 03:21:58.187309 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":275,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.131@KRBTEST.COM" created.
I20260924 03:21:58.188894 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:58.168494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:38241 (local address 127.10.12.130:37141)
0924 03:21:58.169340 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:58.169405 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:58.171773 (+ 2368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:58.172721 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:58.172734 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:58.173538 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:58.174354 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:58.174380 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:58.175561 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:58.175565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:58.176923 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:58.176939 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:58.177805 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:58.177814 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:58.178052 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:58.178734 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:58.178763 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:58.181242 (+ 2479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:58.184484 (+ 3242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:58.184495 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:58.184520 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:58.185022 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:58.185433 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:58.185439 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:58.185443 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:58.185662 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:58.186245 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:58.186255 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:58.186649 (+ 394us) client_negotiation.cc:769] Sending connection context
0924 03:21:58.186924 (+ 275us) client_negotiation.cc:240] Negotiation successful
0924 03:21:58.187524 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":330,"threads_started":1}
I20260924 03:21:58.190661 16247 heartbeater.cc:347] Connected to a master server at 127.10.12.190:38241
I20260924 03:21:58.191033 16247 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:58.191807 16247 heartbeater.cc:511] Master 127.10.12.190:38241 requested a full tablet report, sending...
I20260924 03:21:58.193665 15924 ts_manager.cc:194] Registered new tserver with Master: d4ef28a16be945d7a863f99495896aab (127.10.12.130:41093)
I20260924 03:21:58.194746 15924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.130@KRBTEST.COM'} at 127.10.12.130:37141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
I20260924 03:21:58.217413 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:38241
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:36797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:58.552762 16253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:58.553121 16253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:58.553232 16253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:58.560977 16253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:58.561111 16253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:58.561269 16253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:21:58.571456 16253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:36797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:38241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:58.572991 16253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:58.574361 16253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:58.589203 16258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:58.589280 16259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:58.589252 16261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:58.590375 16253 server_base.cc:1034] running on GCE node
I20260924 03:21:58.591099 16253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:58.592168 16253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:58.593562 16253 hybrid_clock.cc:648] HybridClock initialized: now 1790220118593456 us; error 55 us; skew 500 ppm
I20260924 03:21:58.594115 16253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:58 dist-test-slave-00qs krb5kdc[15879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220118, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:58.597919 16253 init.cc:377] Logged in from keytab as kudu/127.10.12.131@KRBTEST.COM (short username kudu)
I20260924 03:21:58.599527 16253 webserver.cc:460] Webserver started at http://127.10.12.131:41073/ using document root <none> and password file <none>
I20260924 03:21:58.600313 16253 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:58.600430 16253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:58.600701 16253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:58.603924 16253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "9477397ffe71482fa75114db58e685ac"
format_stamp: "Formatted at 2026-09-24 03:21:58 on dist-test-slave-00qs"
server_key: "bd936a264bf6ea1ba00b134e1925837e"
server_key_iv: "87ed5330152800df217d6c865dd105b8"
server_key_version: "encryptionkey@0"
I20260924 03:21:58.604808 16253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "9477397ffe71482fa75114db58e685ac"
format_stamp: "Formatted at 2026-09-24 03:21:58 on dist-test-slave-00qs"
server_key: "bd936a264bf6ea1ba00b134e1925837e"
server_key_iv: "87ed5330152800df217d6c865dd105b8"
server_key_version: "encryptionkey@0"
I20260924 03:21:58.610671 16253 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260924 03:21:58.614707 16268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:58.616357 16253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:21:58.616541 16253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "9477397ffe71482fa75114db58e685ac"
format_stamp: "Formatted at 2026-09-24 03:21:58 on dist-test-slave-00qs"
server_key: "bd936a264bf6ea1ba00b134e1925837e"
server_key_iv: "87ed5330152800df217d6c865dd105b8"
server_key_version: "encryptionkey@0"
I20260924 03:21:58.616760 16253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:58.637302 16253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:58.641886 16253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:58.642186 16253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:58.643260 16253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:21:58.644793 16253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:21:58.644872 16253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:58.645002 16253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:21:58.645053 16253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:58.671530 16113 heartbeater.cc:503] Master 127.10.12.190:38241 was elected leader, sending a full tablet report...
I20260924 03:21:58.678566 16253 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:41819
I20260924 03:21:58.678653 16380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:41819 every 8 connection(s)
I20260924 03:21:58.679929 16253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:21:58.689330 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 16253
I20260924 03:21:58.689514 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:21:58.689972 10290 external_mini_cluster.cc:1428] Setting key 97b9400c61dcc0318a213964330fa954
Sep 24 03:21:58 dist-test-slave-00qs krb5kdc[15879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220118, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:21:58.704794 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:58.682653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:40739 (local address 127.10.12.190:38241)
0924 03:21:58.682890 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:58.682897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:58.684629 (+ 1732us) server_negotiation.cc:407] Connection header received
0924 03:21:58.687038 (+ 2409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:58.687053 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:58.687214 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:58.687364 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:58.690140 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:58.691081 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:58.693166 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:58.693634 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:58.693763 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:58.697676 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:58.697683 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:58.697686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:58.697725 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:58.700296 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:58.701304 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:58.701309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:58.701311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:58.701492 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:58.702147 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:58.702153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:58.702156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:58.702652 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:58.702883 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:58.703858 (+ 975us) server_negotiation.cc:299] Negotiation successful
0924 03:21:58.704333 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 03:21:58.705837 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:58.683112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:38241 (local address 127.10.12.131:40739)
0924 03:21:58.684345 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:58.684410 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:58.686484 (+ 2074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:58.687773 (+ 1289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:58.687793 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:58.688702 (+ 909us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:58.689834 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:58.689867 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:58.691394 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:58.691401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:58.692941 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:58.692966 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:58.693900 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:58.693908 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:58.694163 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:58.694877 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:58.694908 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:58.697346 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:58.700516 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:58.700528 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:58.700547 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:58.701127 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:58.701612 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:58.701615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:58.701618 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:58.701850 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:58.702823 (+ 973us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:58.702830 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:58.703356 (+ 526us) client_negotiation.cc:769] Sending connection context
0924 03:21:58.703651 (+ 295us) client_negotiation.cc:240] Negotiation successful
0924 03:21:58.704388 (+ 737us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":411,"threads_started":1}
I20260924 03:21:58.707361 16381 heartbeater.cc:347] Connected to a master server at 127.10.12.190:38241
I20260924 03:21:58.707679 16381 heartbeater.cc:464] Registering TS with master...
I20260924 03:21:58.708498 16381 heartbeater.cc:511] Master 127.10.12.190:38241 requested a full tablet report, sending...
I20260924 03:21:58.710343 15924 ts_manager.cc:194] Registered new tserver with Master: 9477397ffe71482fa75114db58e685ac (127.10.12.131:41819)
I20260924 03:21:58.711500 15924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.131@KRBTEST.COM'} at 127.10.12.131:40739
I20260924 03:21:58.719914 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/test-admin.keytab.
Sep 24 03:21:58 dist-test-slave-00qs krb5kdc[15879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:58.747184 10290 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 03:21:58.752923 16388 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:21:58.750202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50480 (local address 127.10.12.129:43323)
0924 03:21:58.750929 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:58.750938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:58.750966 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:21:58.751028 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:58.751035 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:58.751260 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:58.751433 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:58.752474 (+ 1041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50480: BlockingRecv error: recv got EOF from 127.0.0.1:50480 (error 108)
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":326,"threads_started":1}
W20260924 03:21:58.753310 10290 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/krb5cc)
Sep 24 03:21:58 dist-test-slave-00qs krb5kdc[15879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:58.755115 10290 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 03:21:58 dist-test-slave-00qs krb5kdc[15879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:58.756697 10290 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 03:21:58 dist-test-slave-00qs krb5kdc[15879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.129@KRBTEST.COM
I20260924 03:21:58.773226 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:58.758511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50492 (local address 127.10.12.129:43323)
0924 03:21:58.758733 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:58.758740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:58.758968 (+ 228us) server_negotiation.cc:407] Connection header received
0924 03:21:58.759138 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:58.759144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:58.759215 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:58.759298 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:58.760354 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:58.761333 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:58.762584 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:58.763182 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:58.763303 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:58.767434 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:58.767453 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:58.767469 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:58.767516 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:58.770326 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:58.771081 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:58.771088 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:58.771091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:58.771225 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:58.771573 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:58.771578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:58.771582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:58.772029 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:58.772325 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:58.772656 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 03:21:58.772926 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 03:21:58.780035 16388 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:21:58.777853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50504 (local address 127.10.12.129:43323)
0924 03:21:58.778295 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:58.778301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:58.778517 (+ 216us) server_negotiation.cc:407] Connection header received
0924 03:21:58.778694 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:58.778700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:58.778773 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:58.778856 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:58.779728 (+ 872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50504: BlockingRecv error: recv got EOF from 127.0.0.1:50504 (error 108)
Metrics: {"server-negotiator.queue_time_us":184}
W20260924 03:21:58.780414 10290 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/krb5cc' of type FILE
Sep 24 03:21:58 dist-test-slave-00qs krb5kdc[15879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:58.781952 10290 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 03:21:58 dist-test-slave-00qs krb5kdc[15879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:58.783263 10290 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 03:21:58 dist-test-slave-00qs krb5kdc[15879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.129@KRBTEST.COM
I20260924 03:21:58.798501 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:58.784751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50514 (local address 127.10.12.129:43323)
0924 03:21:58.784929 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:58.784936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:58.785293 (+ 357us) server_negotiation.cc:407] Connection header received
0924 03:21:58.785475 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:58.785479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:58.785545 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:58.785626 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:58.786617 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:58.787330 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:58.788670 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:58.789100 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:58.789224 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:58.792993 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:58.792999 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:58.793002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:58.793053 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:58.795504 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:58.796297 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:58.796304 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:58.796306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:58.796470 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:58.797081 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:58.797085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:58.797087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:58.797393 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:58.797666 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:58.798018 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 03:21:58.798232 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 03:21:58.804802 16388 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:21:58.801758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50524 (local address 127.10.12.129:43323)
0924 03:21:58.802039 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:58.802047 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:58.803152 (+ 1105us) server_negotiation.cc:407] Connection header received
0924 03:21:58.803364 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:58.803370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:58.803470 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:58.803563 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:58.804556 (+ 993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50524: BlockingRecv error: recv got EOF from 127.0.0.1:50524 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260924 03:21:58.805035 10290 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790220067268388-10290-0/krb5kdc/krb5cc' of type FILE
Sep 24 03:21:58 dist-test-slave-00qs krb5kdc[15879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:58.806677 10290 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 03:21:58 dist-test-slave-00qs krb5kdc[15879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:58.808202 10290 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 03:21:58 dist-test-slave-00qs krb5kdc[15879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.129@KRBTEST.COM
I20260924 03:21:58.824718 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:58.810018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50526 (local address 127.10.12.129:43323)
0924 03:21:58.810417 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:58.810423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:58.810575 (+ 152us) server_negotiation.cc:407] Connection header received
0924 03:21:58.810726 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:58.810729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:58.810817 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:58.810976 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:58.812065 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:58.812816 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:58.814139 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:58.814591 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:58.814697 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:58.818543 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:58.818549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:58.818553 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:58.818596 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:58.821397 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:58.822315 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:58.822322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:58.822325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:58.822617 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:58.823180 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:58.823186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:58.823189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:58.823558 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:58.823846 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:58.824166 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 03:21:58.824371 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260924 03:21:58.826571 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 15985
I20260924 03:21:58.840356 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:58.827397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50536 (local address 127.10.12.129:43323)
0924 03:21:58.827597 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:58.827604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:58.827844 (+ 240us) server_negotiation.cc:407] Connection header received
0924 03:21:58.828051 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:58.828055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:58.828121 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:58.828205 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:58.829272 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:58.829977 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:58.831389 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:58.831851 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:58.832060 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:58.834992 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:58.834998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:58.835001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:58.835042 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:58.837567 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:58.838240 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:58.838244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:58.838246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:58.838389 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:58.838757 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:58.838762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:58.838764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:58.839110 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:58.839316 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:58.839762 (+ 446us) server_negotiation.cc:299] Negotiation successful
0924 03:21:58.840056 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 03:21:58.840973 16108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:58.928103 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 15985
I20260924 03:21:58.941277 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 16119
Sep 24 03:21:58 dist-test-slave-00qs krb5kdc[15879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.130@KRBTEST.COM
I20260924 03:21:58.958719 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:58.942292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45886 (local address 127.10.12.130:41093)
0924 03:21:58.942921 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:58.942929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:58.942957 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:21:58.943019 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:58.943025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:58.943260 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:58.943411 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:58.944498 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:58.945581 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:58.947168 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:58.947875 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:58.948087 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:58.952201 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:58.952216 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:58.952233 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:58.952283 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:58.955457 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:58.956332 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:58.956340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:58.956343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:58.956495 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:58.957056 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:58.957060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:58.957062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:58.957452 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:58.957697 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:58.958119 (+ 422us) server_negotiation.cc:299] Negotiation successful
0924 03:21:58.958396 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":321,"threads_started":1}
I20260924 03:21:58.960098 16242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:59.044004 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 16119
I20260924 03:21:59.056951 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 16253
Sep 24 03:21:59 dist-test-slave-00qs krb5kdc[15879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.131@KRBTEST.COM
I20260924 03:21:59.073208 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:59.057904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59854 (local address 127.10.12.131:41819)
0924 03:21:59.058490 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:59.058497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:59.058526 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:21:59.058590 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:59.058596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:59.058812 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:59.058984 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:59.060085 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:59.061148 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:59.062487 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:59.063104 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:59.063205 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:59.067302 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:59.067317 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:59.067333 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:59.067371 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:59.070170 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:59.071066 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:59.071073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:59.071076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:59.071223 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:59.071647 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:59.071650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:59.071652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:59.072084 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:59.072337 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:59.072637 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 03:21:59.072885 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":274,"threads_started":1}
I20260924 03:21:59.074532 16376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:59.157418 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 16253
I20260924 03:21:59.169610 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 15892
I20260924 03:21:59.170849 15954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:59.274049 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 15892
2026-09-24T03:21:59Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3974 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:21:59 dist-test-slave-00qs krb5kdc[16408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:21:59 dist-test-slave-00qs krb5kdc[16408](info): set up 2 sockets
Sep 24 03:21:59 dist-test-slave-00qs krb5kdc[16408](info): commencing operation
krb5kdc: starting...
W20260924 03:22:00.309202 10290 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:22:00 dist-test-slave-00qs krb5kdc[16408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:22:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:22:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.190.keytab.
Entry for principal oryx/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.190.keytab.
I20260924 03:22:00.532353 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:37301
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.190.keytab
--principal=oryx/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:37301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:22:00.872426 16421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:00.872752 16421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:00.872886 16421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:00.881157 16421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:22:00.881304 16421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:00.881505 16421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:00.881631 16421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:22:00.881697 16421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:22:00.893050 16421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:37301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.190.keytab
--principal=oryx/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:37301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:00.894487 16421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:00.895941 16421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:00.908812 16426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:00.908838 16427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:00.909425 16429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:00.909683 16421 server_base.cc:1034] running on GCE node
I20260924 03:22:00.910504 16421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:00.912165 16421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:00.913429 16421 hybrid_clock.cc:648] HybridClock initialized: now 1790220120913349 us; error 50 us; skew 500 ppm
I20260924 03:22:00.913798 16421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:00 dist-test-slave-00qs krb5kdc[16408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220120, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:00.917203 16421 init.cc:377] Logged in from keytab as oryx/127.10.12.190@KRBTEST.COM (short username oryx)
I20260924 03:22:00.918855 16421 webserver.cc:460] Webserver started at http://127.10.12.190:42145/ using document root <none> and password file <none>
I20260924 03:22:00.919586 16421 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:00.919714 16421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:00.920063 16421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:00.923237 16421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "e9fee3161a524093be6e2e6642cb28c3"
format_stamp: "Formatted at 2026-09-24 03:22:00 on dist-test-slave-00qs"
server_key: "677a08e08d7b84738ca908e7579dc8e1"
server_key_iv: "2a945a4cb82077d4ee7d9a00cdc9b780"
server_key_version: "encryptionkey@0"
I20260924 03:22:00.924136 16421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "e9fee3161a524093be6e2e6642cb28c3"
format_stamp: "Formatted at 2026-09-24 03:22:00 on dist-test-slave-00qs"
server_key: "677a08e08d7b84738ca908e7579dc8e1"
server_key_iv: "2a945a4cb82077d4ee7d9a00cdc9b780"
server_key_version: "encryptionkey@0"
I20260924 03:22:00.930016 16421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:22:00.934111 16436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:00.936388 16421 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260924 03:22:00.936604 16421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "e9fee3161a524093be6e2e6642cb28c3"
format_stamp: "Formatted at 2026-09-24 03:22:00 on dist-test-slave-00qs"
server_key: "677a08e08d7b84738ca908e7579dc8e1"
server_key_iv: "2a945a4cb82077d4ee7d9a00cdc9b780"
server_key_version: "encryptionkey@0"
I20260924 03:22:00.936821 16421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:00.963197 16421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:00.968101 16421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:00.968413 16421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:00.986775 16421 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:37301
I20260924 03:22:00.986855 16487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:37301 every 8 connection(s)
I20260924 03:22:00.988286 16421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:22:00.992354 16488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:00.994092 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 16421
I20260924 03:22:00.994302 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:22:00.995118 10290 external_mini_cluster.cc:1428] Setting key 4d5022caa751ae59a68322cd7db7e2cb
I20260924 03:22:01.007939 16488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3: Bootstrap starting.
Sep 24 03:22:01 dist-test-slave-00qs krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM
I20260924 03:22:01.011627 16488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:01.012926 16488 log.cc:824] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3: Log is configured to *not* fsync() on all Append() calls
I20260924 03:22:01.016258 16488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3: No bootstrap required, opened a new log
I20260924 03:22:01.020825 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:00.997838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35322 (local address 127.10.12.190:37301)
0924 03:22:00.998826 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:00.998846 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:00.998901 (+ 55us) server_negotiation.cc:407] Connection header received
0924 03:22:01.000711 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:01.000740 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:01.001284 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:01.001719 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:01.003411 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:01.004953 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:01.006394 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:01.007215 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:01.007437 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:01.012123 (+ 4686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:01.012133 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:01.012151 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:01.012203 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:01.015565 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:01.016444 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:01.016453 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:01.016462 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:01.016692 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:01.017117 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:01.017123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:01.017126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:01.017900 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:01.018249 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:01.018709 (+ 460us) server_negotiation.cc:299] Negotiation successful
0924 03:22:01.019250 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":399,"threads_started":1}
I20260924 03:22:01.026182 16488 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } }
I20260924 03:22:01.026605 16488 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:01.026696 16488 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fee3161a524093be6e2e6642cb28c3, State: Initialized, Role: FOLLOWER
I20260924 03:22:01.027401 16488 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } }
I20260924 03:22:01.027684 16488 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:22:01.027824 16488 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:22:01.028041 16488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:01.030517 16488 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } }
I20260924 03:22:01.031169 16488 leader_election.cc:302] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9fee3161a524093be6e2e6642cb28c3; no voters:
I20260924 03:22:01.031941 16488 leader_election.cc:288] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:22:01.032100 16493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:22:01.032687 16493 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [term 1 LEADER]: Becoming Leader. State: Replica: e9fee3161a524093be6e2e6642cb28c3, State: Running, Role: LEADER
I20260924 03:22:01.033332 16493 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } }
I20260924 03:22:01.034497 16488 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:22:01.036845 16495 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9fee3161a524093be6e2e6642cb28c3. Latest consensus state: current_term: 1 leader_uuid: "e9fee3161a524093be6e2e6642cb28c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } }
I20260924 03:22:01.036831 16494 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9fee3161a524093be6e2e6642cb28c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } }
I20260924 03:22:01.037294 16495 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:01.037297 16494 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:01.039654 16500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:22:01.045845 16500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:22:01.057052 16500 catalog_manager.cc:1347] Generated new cluster ID: 9da1f5522cb640e3bc32d53489f76eb7
I20260924 03:22:01.057189 16500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:22:01.068980 16500 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:22:01.070403 16500 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:22:01.084095 16500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3: Generated new TSK 0
I20260924 03:22:01.085089 16500 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.129.keytab.
Entry for principal oryx/127.10.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.129.keytab.
I20260924 03:22:01.137609 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:37301
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.129.keytab
--principal=oryx/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:22:01.482117 16514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:01.482438 16514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:01.482564 16514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:01.490113 16514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:01.490247 16514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:01.490391 16514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:22:01.500662 16514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.129.keytab
--principal=oryx/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:37301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:01.502090 16514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:01.503713 16514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:01.518164 16519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:01.518116 16522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:01.518139 16520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:01.520083 16514 server_base.cc:1034] running on GCE node
I20260924 03:22:01.520807 16514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:01.522436 16514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:01.523767 16514 hybrid_clock.cc:648] HybridClock initialized: now 1790220121523704 us; error 61 us; skew 500 ppm
I20260924 03:22:01.524278 16514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:01 dist-test-slave-00qs krb5kdc[16408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220121, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:01.528198 16514 init.cc:377] Logged in from keytab as oryx/127.10.12.129@KRBTEST.COM (short username oryx)
I20260924 03:22:01.530766 16514 webserver.cc:460] Webserver started at http://127.10.12.129:34653/ using document root <none> and password file <none>
I20260924 03:22:01.531865 16514 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:01.532081 16514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:01.532636 16514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:01.536160 16514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "8c17e6da2e044dffbd40657771620fb5"
format_stamp: "Formatted at 2026-09-24 03:22:01 on dist-test-slave-00qs"
server_key: "7fd64b6bd8859b9e4cdb3ac9757fd319"
server_key_iv: "6bd58ac8edf742df9e25019be58eda79"
server_key_version: "encryptionkey@0"
I20260924 03:22:01.536999 16514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "8c17e6da2e044dffbd40657771620fb5"
format_stamp: "Formatted at 2026-09-24 03:22:01 on dist-test-slave-00qs"
server_key: "7fd64b6bd8859b9e4cdb3ac9757fd319"
server_key_iv: "6bd58ac8edf742df9e25019be58eda79"
server_key_version: "encryptionkey@0"
I20260924 03:22:01.542973 16514 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 03:22:01.547927 16529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:01.549576 16514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:22:01.549768 16514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "8c17e6da2e044dffbd40657771620fb5"
format_stamp: "Formatted at 2026-09-24 03:22:01 on dist-test-slave-00qs"
server_key: "7fd64b6bd8859b9e4cdb3ac9757fd319"
server_key_iv: "6bd58ac8edf742df9e25019be58eda79"
server_key_version: "encryptionkey@0"
I20260924 03:22:01.550045 16514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:01.596004 16514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:01.600481 16514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:01.600781 16514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:01.601759 16514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:22:01.603225 16514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:22:01.603318 16514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:01.603431 16514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:22:01.603488 16514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:01.636448 16514 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:38003
I20260924 03:22:01.636499 16641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:38003 every 8 connection(s)
I20260924 03:22:01.637730 16514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:22:01.640888 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 16514
I20260924 03:22:01.641127 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:22:01.641806 10290 external_mini_cluster.cc:1428] Setting key 55fc6141f2afb1b466f110e35f55f933
Sep 24 03:22:01 dist-test-slave-00qs krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220121, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM
I20260924 03:22:01.662107 16644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:01.640573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:54263 (local address 127.10.12.190:37301)
0924 03:22:01.641055 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:01.641063 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:01.642289 (+ 1226us) server_negotiation.cc:407] Connection header received
0924 03:22:01.644172 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:01.644178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:01.644274 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:01.644423 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:01.646346 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:01.647320 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:01.649127 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:01.649716 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:01.649853 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:01.654308 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:01.654314 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:01.654318 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:01.654371 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:01.657764 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:01.659052 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:01.659059 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:01.659062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:01.659244 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:01.659857 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:01.659862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:01.659864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:01.660255 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:01.660409 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:01.661458 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0924 03:22:01.661685 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":228,"threads_started":1}
WARNING: no policy specified for oryx/127.10.12.130@KRBTEST.COM; defaulting to no policy
I20260924 03:22:01.663296 16645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:01.641202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.129:54263)
0924 03:22:01.642113 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:01.642160 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:01.643851 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:01.644682 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:01.644696 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:01.645495 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:01.646144 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:01.646164 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:01.647524 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:01.647528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:01.648955 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:01.648970 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:01.650015 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:01.650023 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:01.650345 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:01.651176 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:01.651204 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:01.654000 (+ 2796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:01.658039 (+ 4039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:01.658051 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:01.658073 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:01.658798 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:01.659376 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:01.659381 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:01.659385 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:01.659674 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:01.660415 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:01.660424 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:01.660953 (+ 529us) client_negotiation.cc:769] Sending connection context
0924 03:22:01.661285 (+ 332us) client_negotiation.cc:240] Negotiation successful
0924 03:22:01.661951 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":403,"threads_started":1}
I20260924 03:22:01.665161 16642 heartbeater.cc:347] Connected to a master server at 127.10.12.190:37301
I20260924 03:22:01.665508 16642 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.12.130@KRBTEST.COM" created.
I20260924 03:22:01.666431 16642 heartbeater.cc:511] Master 127.10.12.190:37301 requested a full tablet report, sending...
I20260924 03:22:01.669135 16453 ts_manager.cc:194] Registered new tserver with Master: 8c17e6da2e044dffbd40657771620fb5 (127.10.12.129:38003)
I20260924 03:22:01.671120 16453 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.12.129@KRBTEST.COM'} at 127.10.12.129:54263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.130.keytab.
Entry for principal oryx/127.10.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.130.keytab.
I20260924 03:22:01.697328 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:37301
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.130.keytab
--principal=oryx/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:22:02.054513 16648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:02.054840 16648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:02.055083 16648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:02.062856 16648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:02.063024 16648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:02.063189 16648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:22:02.073501 16648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.130.keytab
--principal=oryx/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:37301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:02.074906 16648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:02.076320 16648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:02.090488 16656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:02.090536 16654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:02.090447 16653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:02.091115 16648 server_base.cc:1034] running on GCE node
I20260924 03:22:02.091797 16648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:02.092854 16648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:02.094135 16648 hybrid_clock.cc:648] HybridClock initialized: now 1790220122094071 us; error 57 us; skew 500 ppm
I20260924 03:22:02.094496 16648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:02 dist-test-slave-00qs krb5kdc[16408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220122, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:02.102404 16648 init.cc:377] Logged in from keytab as oryx/127.10.12.130@KRBTEST.COM (short username oryx)
I20260924 03:22:02.104357 16648 webserver.cc:460] Webserver started at http://127.10.12.130:36451/ using document root <none> and password file <none>
I20260924 03:22:02.105556 16648 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:02.105855 16648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:02.106269 16648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:02.109722 16648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "3cb91631ae5147c6aca234d1afe3b4cc"
format_stamp: "Formatted at 2026-09-24 03:22:02 on dist-test-slave-00qs"
server_key: "84cc17252850f93a7707f10f96dd1a94"
server_key_iv: "cc8c9d11b4fafa8b31e1eb5241cb27fc"
server_key_version: "encryptionkey@0"
I20260924 03:22:02.110600 16648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "3cb91631ae5147c6aca234d1afe3b4cc"
format_stamp: "Formatted at 2026-09-24 03:22:02 on dist-test-slave-00qs"
server_key: "84cc17252850f93a7707f10f96dd1a94"
server_key_iv: "cc8c9d11b4fafa8b31e1eb5241cb27fc"
server_key_version: "encryptionkey@0"
I20260924 03:22:02.116400 16648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 03:22:02.120308 16663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:02.122022 16648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:22:02.122211 16648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "3cb91631ae5147c6aca234d1afe3b4cc"
format_stamp: "Formatted at 2026-09-24 03:22:02 on dist-test-slave-00qs"
server_key: "84cc17252850f93a7707f10f96dd1a94"
server_key_iv: "cc8c9d11b4fafa8b31e1eb5241cb27fc"
server_key_version: "encryptionkey@0"
I20260924 03:22:02.122423 16648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:02.166427 16648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:02.171862 16648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:02.172250 16648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:02.173230 16648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:22:02.174671 16648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:22:02.174755 16648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:02.174873 16648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:22:02.174923 16648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:02.208681 16648 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:39379
I20260924 03:22:02.208732 16775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:39379 every 8 connection(s)
I20260924 03:22:02.209990 16648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:22:02.211880 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 16648
I20260924 03:22:02.212077 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:22:02.212558 10290 external_mini_cluster.cc:1428] Setting key aee63d0f027ad3105d2ddb25bcf730be
Sep 24 03:22:02 dist-test-slave-00qs krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220122, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.10.12.131@KRBTEST.COM; defaulting to no policy
I20260924 03:22:02.233566 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:02.212706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:60821 (local address 127.10.12.190:37301)
0924 03:22:02.213173 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:02.213180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:02.214368 (+ 1188us) server_negotiation.cc:407] Connection header received
0924 03:22:02.216279 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:02.216285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:02.216372 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:02.216472 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:02.218121 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:02.218850 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:02.220929 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:02.221380 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:02.221503 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:02.226481 (+ 4978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:02.226489 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:02.226493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:02.226545 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:02.229754 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:02.230734 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:02.230738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:02.230741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:02.230905 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:02.231537 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:02.231541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:02.231544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:02.231904 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:02.232167 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:02.232989 (+ 822us) server_negotiation.cc:299] Negotiation successful
0924 03:22:02.233255 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":222,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.12.131@KRBTEST.COM" created.
I20260924 03:22:02.234673 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:02.213311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.130:60821)
0924 03:22:02.214190 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:02.214246 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:02.215982 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:02.216743 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:02.216754 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:02.217297 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:02.217920 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:02.217938 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:02.219019 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:02.219023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:02.220743 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:02.220761 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:02.221630 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:02.221638 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:02.221888 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:02.222703 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:02.222729 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:02.226057 (+ 3328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:02.229996 (+ 3939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:02.230006 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:02.230025 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:02.230579 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:02.231076 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:02.231083 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:02.231088 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:02.231360 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:02.232146 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:02.232153 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:02.232562 (+ 409us) client_negotiation.cc:769] Sending connection context
0924 03:22:02.232818 (+ 256us) client_negotiation.cc:240] Negotiation successful
0924 03:22:02.233323 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":324,"threads_started":1}
I20260924 03:22:02.236796 16776 heartbeater.cc:347] Connected to a master server at 127.10.12.190:37301
I20260924 03:22:02.237181 16776 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:02.238134 16776 heartbeater.cc:511] Master 127.10.12.190:37301 requested a full tablet report, sending...
I20260924 03:22:02.240285 16453 ts_manager.cc:194] Registered new tserver with Master: 3cb91631ae5147c6aca234d1afe3b4cc (127.10.12.130:39379)
I20260924 03:22:02.241479 16453 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.12.130@KRBTEST.COM'} at 127.10.12.130:60821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.131.keytab.
Entry for principal oryx/127.10.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.131.keytab.
I20260924 03:22:02.265221 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:37301
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.131.keytab
--principal=oryx/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:22:02.597682 16782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:02.597999 16782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:02.598136 16782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:02.605625 16782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:02.605760 16782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:02.605902 16782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:22:02.616031 16782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.131.keytab
--principal=oryx/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:37301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:02.617426 16782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:02.618800 16782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:02.632638 16787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:02.632586 16788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:02.632560 16790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:02.633610 16782 server_base.cc:1034] running on GCE node
I20260924 03:22:02.634289 16782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:02.635352 16782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:02.636580 16782 hybrid_clock.cc:648] HybridClock initialized: now 1790220122636524 us; error 46 us; skew 500 ppm
I20260924 03:22:02.636969 16782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:02 dist-test-slave-00qs krb5kdc[16408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220122, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:02.641149 16782 init.cc:377] Logged in from keytab as oryx/127.10.12.131@KRBTEST.COM (short username oryx)
I20260924 03:22:02.642784 16782 webserver.cc:460] Webserver started at http://127.10.12.131:44201/ using document root <none> and password file <none>
I20260924 03:22:02.643462 16782 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:02.643565 16782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:02.643813 16782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:02.646901 16782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "82c9a19bc3f44bf0bf0739243ee16066"
format_stamp: "Formatted at 2026-09-24 03:22:02 on dist-test-slave-00qs"
server_key: "ea30e4073bbaf509c08c8d5c2583ccbf"
server_key_iv: "08718d0528896517a96030f6902fdd27"
server_key_version: "encryptionkey@0"
I20260924 03:22:02.647722 16782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "82c9a19bc3f44bf0bf0739243ee16066"
format_stamp: "Formatted at 2026-09-24 03:22:02 on dist-test-slave-00qs"
server_key: "ea30e4073bbaf509c08c8d5c2583ccbf"
server_key_iv: "08718d0528896517a96030f6902fdd27"
server_key_version: "encryptionkey@0"
I20260924 03:22:02.653489 16782 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 03:22:02.657397 16797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:02.658960 16782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:22:02.659126 16782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "82c9a19bc3f44bf0bf0739243ee16066"
format_stamp: "Formatted at 2026-09-24 03:22:02 on dist-test-slave-00qs"
server_key: "ea30e4073bbaf509c08c8d5c2583ccbf"
server_key_iv: "08718d0528896517a96030f6902fdd27"
server_key_version: "encryptionkey@0"
I20260924 03:22:02.659325 16782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:02.675310 16642 heartbeater.cc:503] Master 127.10.12.190:37301 was elected leader, sending a full tablet report...
I20260924 03:22:02.683068 16782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:02.688078 16782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:02.688345 16782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:02.689316 16782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:22:02.690716 16782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:22:02.690817 16782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:02.690912 16782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:22:02.690954 16782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:02.724627 16782 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:46457
I20260924 03:22:02.724704 16909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:46457 every 8 connection(s)
I20260924 03:22:02.725971 16782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:22:02.726712 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 16782
I20260924 03:22:02.727002 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:22:02.727604 10290 external_mini_cluster.cc:1428] Setting key c01ace2d1190df23eaa6a7760fa9e695
Sep 24 03:22:02 dist-test-slave-00qs krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220122, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM
I20260924 03:22:02.747587 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:02.728735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:45793 (local address 127.10.12.190:37301)
0924 03:22:02.728952 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:02.728958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:02.730169 (+ 1211us) server_negotiation.cc:407] Connection header received
0924 03:22:02.732220 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:02.732225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:02.732331 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:02.732421 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:02.734027 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:02.734763 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:02.736595 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:02.737014 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:02.737128 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:02.740927 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:02.740933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:02.740937 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:02.740975 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:02.744291 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:02.745102 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:02.745106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:02.745108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:02.745228 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:02.745708 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:02.745712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:02.745714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:02.746075 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:02.746216 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:02.747012 (+ 796us) server_negotiation.cc:299] Negotiation successful
0924 03:22:02.747256 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 03:22:02.748624 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:02.729218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.131:45793)
0924 03:22:02.730020 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:02.730063 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:02.731884 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:02.732685 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:02.732694 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:02.733236 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:02.733844 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:02.733863 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:02.734946 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:02.734950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:02.736415 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:02.736430 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:02.737251 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:02.737259 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:02.737511 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:02.738260 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:02.738284 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:02.740647 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:02.744461 (+ 3814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:02.744471 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:02.744488 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:02.744964 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:02.745336 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:02.745341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:02.745344 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:02.745587 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:02.746211 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:02.746220 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:02.746612 (+ 392us) client_negotiation.cc:769] Sending connection context
0924 03:22:02.746858 (+ 246us) client_negotiation.cc:240] Negotiation successful
0924 03:22:02.747435 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":307,"threads_started":1}
I20260924 03:22:02.750198 16910 heartbeater.cc:347] Connected to a master server at 127.10.12.190:37301
I20260924 03:22:02.750663 16910 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:02.751442 16910 heartbeater.cc:511] Master 127.10.12.190:37301 requested a full tablet report, sending...
I20260924 03:22:02.752972 16453 ts_manager.cc:194] Registered new tserver with Master: 82c9a19bc3f44bf0bf0739243ee16066 (127.10.12.131:46457)
I20260924 03:22:02.753834 16453 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.12.131@KRBTEST.COM'} at 127.10.12.131:45793
I20260924 03:22:02.756542 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.12.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.12.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.12.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.189.keytab.
Entry for principal oryx/127.10.12.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.189.keytab.
I20260924 03:22:02.805365 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.189:33891
--webserver_interface=127.10.12.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.189.keytab
--principal=oryx/127.10.12.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:37301,127.10.12.189:33891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:22:03.139154 16915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:03.139518 16915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:03.139640 16915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:03.147379 16915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:22:03.147513 16915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:03.147562 16915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:03.147637 16915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:22:03.147694 16915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:22:03.158157 16915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:37301,127.10.12.189:33891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.189.keytab
--principal=oryx/127.10.12.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.189:33891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.12.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
W20260924 03:22:03.158542 16915 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.10.12.190:37301,127.10.12.189:33891'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 03:22:03.159668 16915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:03.161095 16915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:03.173328 16923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:03.173388 16921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:03.173388 16920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:03.174218 16915 server_base.cc:1034] running on GCE node
I20260924 03:22:03.175310 16915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:03.176994 16915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:03.178246 16915 hybrid_clock.cc:648] HybridClock initialized: now 1790220123178168 us; error 53 us; skew 500 ppm
I20260924 03:22:03.178588 16915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:03 dist-test-slave-00qs krb5kdc[16408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220123, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:03.182232 16915 init.cc:377] Logged in from keytab as oryx/127.10.12.189@KRBTEST.COM (short username oryx)
I20260924 03:22:03.184151 16915 webserver.cc:460] Webserver started at http://127.10.12.189:46129/ using document root <none> and password file <none>
I20260924 03:22:03.184937 16915 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:03.185050 16915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:03.185374 16915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:03.188730 16915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/data/instance:
uuid: "85aaf56abb0d4474b9dd4bb4171f557c"
format_stamp: "Formatted at 2026-09-24 03:22:03 on dist-test-slave-00qs"
server_key: "ba0cc55b005bcc8e60ffba9c9747c8da"
server_key_iv: "f1c1ff9040d88594066de7cc6ef5f121"
server_key_version: "encryptionkey@0"
I20260924 03:22:03.189663 16915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/wal/instance:
uuid: "85aaf56abb0d4474b9dd4bb4171f557c"
format_stamp: "Formatted at 2026-09-24 03:22:03 on dist-test-slave-00qs"
server_key: "ba0cc55b005bcc8e60ffba9c9747c8da"
server_key_iv: "f1c1ff9040d88594066de7cc6ef5f121"
server_key_version: "encryptionkey@0"
I20260924 03:22:03.195797 16915 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 03:22:03.200089 16930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:03.201731 16915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:22:03.201948 16915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/wal
uuid: "85aaf56abb0d4474b9dd4bb4171f557c"
format_stamp: "Formatted at 2026-09-24 03:22:03 on dist-test-slave-00qs"
server_key: "ba0cc55b005bcc8e60ffba9c9747c8da"
server_key_iv: "f1c1ff9040d88594066de7cc6ef5f121"
server_key_version: "encryptionkey@0"
I20260924 03:22:03.202224 16915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:03.227209 16915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:03.232651 16915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:03.232931 16915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:03.245024 16776 heartbeater.cc:503] Master 127.10.12.190:37301 was elected leader, sending a full tablet report...
I20260924 03:22:03.251652 16915 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.189:33891
I20260924 03:22:03.251669 16981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.189:33891 every 8 connection(s)
I20260924 03:22:03.253103 16915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/data/info.pb
I20260924 03:22:03.257146 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 16915
I20260924 03:22:03.257660 16982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:03.266815 16982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:22:03 dist-test-slave-00qs krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.12.189@KRBTEST.COM
Sep 24 03:22:03 dist-test-slave-00qs krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220123, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.189@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM
I20260924 03:22:03.282768 16985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:03.259598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49946 (local address 127.10.12.189:33891)
0924 03:22:03.260410 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:03.260426 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:03.260469 (+ 43us) server_negotiation.cc:407] Connection header received
0924 03:22:03.262498 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:03.262526 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:03.262997 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:03.263465 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:03.265081 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.266411 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.267761 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.268533 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.268723 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:03.274246 (+ 5523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:03.274254 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:03.274265 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:03.274311 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:03.277977 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:03.278987 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:03.278997 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:03.279008 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:03.279237 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:03.279713 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:03.279719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:03.279721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:03.280480 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:03.280760 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:03.281255 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 03:22:03.281691 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":369,"threads_started":1}
I20260924 03:22:03.286360 16987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:03.269515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.0.0.1:35338)
0924 03:22:03.270334 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:03.270389 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:03.270566 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:03.270994 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:03.271004 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:03.271873 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:03.272253 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:03.272272 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.273664 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.273670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:03.274999 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:03.275013 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.276268 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.276280 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:03.276545 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:03.277408 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:03.277433 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:03.280308 (+ 2875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:03.283501 (+ 3193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:03.283514 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:03.283521 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:03.284055 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:03.284443 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:03.284448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:03.284452 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:03.284576 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:03.285193 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:03.285203 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:03.285369 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 03:22:03.285530 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 03:22:03.285900 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":390,"threads_started":1}
I20260924 03:22:03.286288 16986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:03.269496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35338 (local address 127.10.12.190:37301)
0924 03:22:03.269996 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:03.270001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:03.270499 (+ 498us) server_negotiation.cc:407] Connection header received
0924 03:22:03.270697 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:03.270701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:03.270790 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:03.270882 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:03.272438 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.273500 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.275156 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.275790 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.275934 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:03.280534 (+ 4600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:03.280540 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:03.280543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:03.280579 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:03.283350 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:03.284187 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:03.284191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:03.284194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:03.284331 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:03.284703 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:03.284707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:03.284709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:03.285033 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:03.285172 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:03.285665 (+ 493us) server_negotiation.cc:299] Negotiation successful
0924 03:22:03.285911 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":234,"threads_started":1}
I20260924 03:22:03.288321 16982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:22:03 dist-test-slave-00qs krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220123, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.189@KRBTEST.COM for oryx/127.10.12.189@KRBTEST.COM
I20260924 03:22:03.302104 16987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:03.288976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:33891 (local address 127.0.0.1:49962)
0924 03:22:03.289378 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:03.289407 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:03.289600 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:03.289986 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:03.289990 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:03.290352 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:03.290494 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:03.290504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.291603 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.291607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:03.292604 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:03.292618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.293431 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.293436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:03.293612 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:03.294314 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:03.294333 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:03.296572 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:03.299657 (+ 3085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:03.299662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:03.299664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:03.300184 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:03.300563 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:03.300567 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:03.300569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:03.300704 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:03.301304 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:03.301308 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:03.301439 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 03:22:03.301561 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 03:22:03.301805 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260924 03:22:03.302407 16985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:03.289204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49962 (local address 127.10.12.189:33891)
0924 03:22:03.289422 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:03.289426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:03.289545 (+ 119us) server_negotiation.cc:407] Connection header received
0924 03:22:03.289729 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:03.289734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:03.289811 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:03.289981 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:03.290668 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.291446 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.292762 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.293251 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.293364 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:03.296806 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:03.296812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:03.296815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:03.296849 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:03.299465 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:03.300306 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:03.300310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:03.300313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:03.300447 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:03.300818 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:03.300822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:03.300824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:03.301186 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:03.301301 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:03.301660 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 03:22:03.302069 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":162,"server-negotiator.queue_time_us":51}
I20260924 03:22:03.307404 16982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c: Bootstrap starting.
I20260924 03:22:03.310976 16982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:03.312134 16982 log.cc:824] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:22:03.316067 16982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c: No bootstrap required, opened a new log
I20260924 03:22:03.323441 16982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } }
I20260924 03:22:03.323843 16982 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:03.323995 16982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85aaf56abb0d4474b9dd4bb4171f557c, State: Initialized, Role: FOLLOWER
I20260924 03:22:03.324643 16982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } }
I20260924 03:22:03.325757 16989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } } }
I20260924 03:22:03.326149 16989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:22:03.326788 16982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [sys.catalog]: configured and running, proceeding with master startup.
W20260924 03:22:03.336970 17000 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:22:03.337250 17000 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:22:03.341337 16915 master_runner.cc:428] Detected that this master 85aaf56abb0d4474b9dd4bb4171f557c is joining an existing cluster
I20260924 03:22:03.341450 16915 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.12.189:33891
WARNING: no policy specified for oryx/127.10.12.188@KRBTEST.COM; defaulting to no policy
I20260924 03:22:03.364827 16986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:03.347003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.10.12.190:37301)
0924 03:22:03.347174 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:03.347178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:03.347837 (+ 659us) server_negotiation.cc:407] Connection header received
0924 03:22:03.348301 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:03.348307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:03.348423 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:03.348527 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:03.350208 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.351473 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.354064 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.354816 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.354923 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:03.357688 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:03.357696 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:03.357701 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:03.357745 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:03.361260 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:03.362079 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:03.362085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:03.362089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:03.362255 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:03.362935 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:03.362941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:03.362944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:03.363319 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:03.363467 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:03.364296 (+ 829us) server_negotiation.cc:299] Negotiation successful
0924 03:22:03.364559 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 03:22:03.366077 17007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:03.347042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.0.0.1:35348)
0924 03:22:03.347722 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:03.347741 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:03.347978 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:03.348586 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:03.348591 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:03.349488 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:03.349972 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:03.349989 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.351720 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.351726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:03.353784 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:03.353813 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.355296 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.355302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:03.355475 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:03.356138 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:03.356159 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:03.357430 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:03.361403 (+ 3973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:03.361409 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:03.361413 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:03.361933 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:03.362398 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:03.362403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:03.362406 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:03.362802 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:03.363651 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:03.363659 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:03.363928 (+ 269us) client_negotiation.cc:769] Sending connection context
0924 03:22:03.365174 (+ 1246us) client_negotiation.cc:240] Negotiation successful
0924 03:22:03.365674 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":384,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.12.188@KRBTEST.COM" created.
I20260924 03:22:03.377964 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:03.348053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:33891 (local address 127.0.0.1:49970)
0924 03:22:03.355302 (+ 7249us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:03.355329 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:03.355675 (+ 346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:03.356480 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:03.356484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:03.357135 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:03.357498 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:03.357508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.359179 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.359187 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:03.361059 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:03.361077 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.362323 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.362329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:03.362517 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:03.363281 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:03.363304 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:03.364384 (+ 1080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:03.374486 (+ 10102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:03.374492 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:03.374497 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:03.375104 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:03.375622 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:03.375626 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:03.375630 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:03.375784 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:03.376773 (+ 989us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:03.376783 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:03.376956 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 03:22:03.377111 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 03:22:03.377437 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7078,"thread_start_us":7351,"threads_started":1}
I20260924 03:22:03.378919 16985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:03.348026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49970 (local address 127.10.12.189:33891)
0924 03:22:03.348178 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:03.348182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:03.355402 (+ 7220us) server_negotiation.cc:407] Connection header received
0924 03:22:03.356126 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:03.356131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:03.356202 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:03.356268 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:03.357641 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.358968 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.361231 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.361907 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.362012 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:03.364583 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:03.364590 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:03.364596 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:03.364634 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:03.374253 (+ 9619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:03.375244 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:03.375250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:03.375254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:03.375476 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:03.375947 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:03.376005 (+ 58us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:03.376009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:03.376531 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:03.376716 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:03.378338 (+ 1622us) server_negotiation.cc:299] Negotiation successful
0924 03:22:03.378572 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 03:22:03 dist-test-slave-00qs krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220120, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.190@KRBTEST.COM for oryx/127.10.12.189@KRBTEST.COM
I20260924 03:22:03.399394 16985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:03.372723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49982 (local address 127.10.12.189:33891)
0924 03:22:03.379147 (+ 6424us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:03.379153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:03.381089 (+ 1936us) server_negotiation.cc:407] Connection header received
0924 03:22:03.381581 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:03.381587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:03.381730 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:03.381830 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:03.383735 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.384824 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.386971 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.387762 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.387945 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:03.391689 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:03.391699 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:03.391703 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:03.391744 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:03.395429 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:03.396474 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:03.396479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:03.396483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:03.396642 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:03.397043 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:03.397047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:03.397049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:03.397458 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:03.397619 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:03.398650 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0924 03:22:03.398960 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6320,"thread_start_us":7474,"threads_started":1}
I20260924 03:22:03.399430 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:03.372374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:33891 (local address 127.0.0.1:49982)
0924 03:22:03.380932 (+ 8558us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:03.380986 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:03.381271 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:03.382154 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:03.382162 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:03.383127 (+ 965us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:03.383480 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:03.383496 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.385018 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.385025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:03.386735 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:03.386758 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.388020 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.388028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:03.388266 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:03.388979 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:03.388999 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:03.391431 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:03.395651 (+ 4220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:03.395675 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:03.395684 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:03.396327 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:03.396763 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:03.396769 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:03.396773 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:03.396923 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:03.397603 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:03.397613 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:03.397819 (+ 206us) client_negotiation.cc:769] Sending connection context
0924 03:22:03.398412 (+ 593us) client_negotiation.cc:240] Negotiation successful
0924 03:22:03.398984 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8247,"thread_start_us":9150,"threads_started":1}
I20260924 03:22:03.402516 16453 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.12.189:33891: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c"
member_type: NON_VOTER
last_known_addr {
host: "127.10.12.189"
port: 33891
}
attrs {
promote: true
}
}
dest_uuid: "e9fee3161a524093be6e2e6642cb28c3"
cas_config_opid_index: -1
I20260924 03:22:03.403931 16453 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: NON_VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.12.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.188.keytab.
Entry for principal oryx/127.10.12.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.188.keytab.
I20260924 03:22:03.409132 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.188:36159
--webserver_interface=127.10.12.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.188.keytab
--principal=oryx/127.10.12.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:37301,127.10.12.189:33891,127.10.12.188:36159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:22:03.409502 16957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:03.418885 16957 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 1 FOLLOWER]: Refusing update from remote peer e9fee3161a524093be6e2e6642cb28c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 03:22:03.418820 17014 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 85aaf56abb0d4474b9dd4bb4171f557c (127.10.12.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: NON_VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: true } } }
I20260924 03:22:03.424062 17013 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: NON_VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:22:03.430662 17014 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.12.189:33891
I20260924 03:22:03.431618 16915 master.cc:537] Master@127.10.12.189:33891 shutting down...
I20260924 03:22:03.432498 17016 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e9fee3161a524093be6e2e6642cb28c3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: NON_VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: true } } }
I20260924 03:22:03.432852 17016 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:03.452412 16957 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 85aaf56abb0d4474b9dd4bb4171f557c (127.10.12.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: NON_VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: true } } }
I20260924 03:22:03.468924 16989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9fee3161a524093be6e2e6642cb28c3. Latest consensus state: current_term: 1 leader_uuid: "e9fee3161a524093be6e2e6642cb28c3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: NON_VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: true } } }
I20260924 03:22:03.469223 16989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [sys.catalog]: This master's current role is: LEARNER
I20260924 03:22:03.469981 17020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9fee3161a524093be6e2e6642cb28c3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: NON_VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: true } } }
I20260924 03:22:03.470248 17020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [sys.catalog]: This master's current role is: LEARNER
I20260924 03:22:03.470048 17024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9fee3161a524093be6e2e6642cb28c3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: NON_VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: true } } }
I20260924 03:22:03.472448 17024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [sys.catalog]: This master's current role is: LEARNER
I20260924 03:22:03.474201 16915 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 1 LEARNER]: Raft consensus shutting down.
I20260924 03:22:03.474666 16915 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 1 LEARNER]: Raft consensus is shut down!
I20260924 03:22:03.474833 16915 tablet_replica.cc:330] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c: stopping tablet replica
I20260924 03:22:03.492267 16915 master.cc:552] Master@127.10.12.189:33891 shutdown complete.
I20260924 03:22:03.492414 16915 master_runner.cc:305] Clearing existing system tablet
I20260924 03:22:03.492770 16915 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:03.492862 16915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:03.495075 16915 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:22:03.496519 17026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:03.496855 16915 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:03.497117 16915 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:22:03.497210 16915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/wal
uuid: "85aaf56abb0d4474b9dd4bb4171f557c"
format_stamp: "Formatted at 2026-09-24 03:22:03 on dist-test-slave-00qs"
server_key: "ba0cc55b005bcc8e60ffba9c9747c8da"
server_key_iv: "f1c1ff9040d88594066de7cc6ef5f121"
server_key_version: "encryptionkey@0"
I20260924 03:22:03.500602 16915 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:22:03.505528 16915 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 03:22:03.505722 16915 log.cc:1196] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 03:22:03.506191 16915 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c: Deleting consensus metadata
I20260924 03:22:03.506444 16915 master_runner.cc:315] Copying system tablet from 127.10.12.190:37301
I20260924 03:22:03.509644 16915 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c: tablet copy: Beginning tablet copy session from remote peer at address 127.10.12.190:37301
I20260924 03:22:03.524039 16986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:03.510415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35360 (local address 127.10.12.190:37301)
0924 03:22:03.510610 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:03.510617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:03.512601 (+ 1984us) server_negotiation.cc:407] Connection header received
0924 03:22:03.512792 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:03.512796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:03.512864 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:03.512950 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:03.513787 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.514826 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.516121 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.516531 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.516641 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:03.518632 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:03.518637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:03.518640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:03.518674 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:03.521243 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:03.521892 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:03.521896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:03.521899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:03.522023 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:03.522374 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:03.522379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:03.522382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:03.522684 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:03.522821 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:03.523417 (+ 596us) server_negotiation.cc:299] Negotiation successful
0924 03:22:03.523637 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:22:03.524117 17032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:03.510354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.0.0.1:35360)
0924 03:22:03.512478 (+ 2124us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:03.512498 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:03.512657 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:03.512993 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:03.512998 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:03.513414 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:03.513639 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:03.513649 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.514987 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.514993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:03.515918 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:03.515929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.516684 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.516690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:03.516889 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:03.517569 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:03.517587 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:03.518393 (+ 806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:03.521395 (+ 3002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:03.521401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:03.521405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:03.521773 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:03.522131 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:03.522136 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:03.522140 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:03.522280 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:03.522806 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:03.522811 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:03.523072 (+ 261us) client_negotiation.cc:769] Sending connection context
0924 03:22:03.523231 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 03:22:03.523580 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2004,"thread_start_us":1814,"threads_started":1}
I20260924 03:22:03.525420 16473 tablet_copy_service.cc:140] P e9fee3161a524093be6e2e6642cb28c3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 85aaf56abb0d4474b9dd4bb4171f557c ({username='oryx', principal='oryx/127.10.12.189@KRBTEST.COM'} at 127.0.0.1:35360)
I20260924 03:22:03.525709 16473 tablet_copy_service.cc:161] P e9fee3161a524093be6e2e6642cb28c3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 85aaf56abb0d4474b9dd4bb4171f557c at {username='oryx', principal='oryx/127.10.12.189@KRBTEST.COM'} at 127.0.0.1:35360: session id = 85aaf56abb0d4474b9dd4bb4171f557c-00000000000000000000000000000000
I20260924 03:22:03.529150 16473 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 03:22:03.531256 16915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 03:22:03.536839 16915 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c: tablet copy: Starting download of 0 data blocks...
I20260924 03:22:03.537189 16915 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c: tablet copy: Starting download of 1 WAL segments...
I20260924 03:22:03.539750 16915 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 03:22:03.543798 16473 tablet_copy_service.cc:342] P e9fee3161a524093be6e2e6642cb28c3: Request end of tablet copy session 85aaf56abb0d4474b9dd4bb4171f557c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.12.189@KRBTEST.COM'} at 127.0.0.1:35360
I20260924 03:22:03.544046 16473 tablet_copy_service.cc:433] P e9fee3161a524093be6e2e6642cb28c3: ending tablet copy session 85aaf56abb0d4474b9dd4bb4171f557c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 85aaf56abb0d4474b9dd4bb4171f557c
I20260924 03:22:03.554240 16915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:03.556874 17033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:03.564030 17034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:03.568415 16915 server_base.cc:1034] running on GCE node
W20260924 03:22:03.569041 17036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:03.569535 16915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:03.570281 16915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:03.571620 16915 hybrid_clock.cc:648] HybridClock initialized: now 1790220123571550 us; error 39 us; skew 500 ppm
I20260924 03:22:03.571882 16915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:03 dist-test-slave-00qs krb5kdc[16408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220123, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:03.573851 16915 init.cc:377] Logged in from keytab as oryx/127.10.12.189@KRBTEST.COM (short username oryx)
I20260924 03:22:03.575135 16915 webserver.cc:460] Webserver started at http://127.10.12.189:39515/ using document root <none> and password file <none>
I20260924 03:22:03.575459 16915 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:03.575537 16915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:03.578174 16915 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260924 03:22:03.579875 17043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:03.580448 16915 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:22:03.580575 16915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/wal
uuid: "85aaf56abb0d4474b9dd4bb4171f557c"
format_stamp: "Formatted at 2026-09-24 03:22:03 on dist-test-slave-00qs"
server_key: "ba0cc55b005bcc8e60ffba9c9747c8da"
server_key_iv: "f1c1ff9040d88594066de7cc6ef5f121"
server_key_version: "encryptionkey@0"
I20260924 03:22:03.580734 16915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:03.596588 16915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:03.600059 16915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:03.633111 16915 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.189:33891
I20260924 03:22:03.633139 17094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.189:33891 every 8 connection(s)
I20260924 03:22:03.634068 16915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-1/data/info.pb
I20260924 03:22:03.645147 17095 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:22:03.648053 17095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c: Bootstrap starting.
I20260924 03:22:03.671588 17095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:22:03.672487 17095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c: Bootstrap complete.
I20260924 03:22:03.673724 17095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: NON_VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: true } }
I20260924 03:22:03.674165 17095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85aaf56abb0d4474b9dd4bb4171f557c, State: Initialized, Role: LEARNER
I20260924 03:22:03.674487 17095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: NON_VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: true } }
I20260924 03:22:03.682247 17095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:22:03.684042 17098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: NON_VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: true } } }
I20260924 03:22:03.684531 17098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [sys.catalog]: This master's current role is: LEARNER
I20260924 03:22:03.756989 16910 heartbeater.cc:503] Master 127.10.12.190:37301 was elected leader, sending a full tablet report...
I20260924 03:22:03.765002 17109 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 03:22:03.850127 17014 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9fee3161a524093be6e2e6642cb28c3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: NON_VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: true } } }
I20260924 03:22:03.850417 17014 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:03.868366 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:03.850336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:33891 (local address 127.0.0.1:49986)
0924 03:22:03.850608 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:03.850628 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:03.850750 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:03.851682 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:03.851687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:03.852203 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:03.852410 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:03.852421 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.854542 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.854548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:03.855927 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:03.856236 (+ 309us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.857789 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.857798 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:03.858007 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:03.858808 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:03.858834 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:03.859937 (+ 1103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:03.864884 (+ 4947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:03.864890 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:03.864895 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:03.865454 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:03.865932 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:03.865937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:03.865940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:03.866106 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:03.867339 (+ 1233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:03.867344 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:03.867532 (+ 188us) client_negotiation.cc:769] Sending connection context
0924 03:22:03.867656 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 03:22:03.867980 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260924 03:22:03.869606 17110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:03.850513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49986 (local address 127.10.12.189:33891)
0924 03:22:03.851310 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:03.851316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:03.851336 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:22:03.851415 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:03.851421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:03.851509 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:03.852556 (+ 1047us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:03.852784 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.854354 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.856656 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:03.857412 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:03.860235 (+ 2823us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:03.860591 (+ 356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:03.860598 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:03.860603 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:03.860638 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:03.864673 (+ 4035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:03.865632 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:03.865637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:03.865641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:03.865798 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:03.866383 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:03.866391 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:03.866394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:03.867178 (+ 784us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:03.868793 (+ 1615us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:03.868958 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 03:22:03.869190 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":489,"threads_started":1}
I20260924 03:22:03.870752 17098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9fee3161a524093be6e2e6642cb28c3. Latest consensus state: current_term: 1 leader_uuid: "e9fee3161a524093be6e2e6642cb28c3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: NON_VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: true } } }
I20260924 03:22:03.871341 17098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [sys.catalog]: This master's current role is: LEARNER
I20260924 03:22:03.872449 17016 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3: attempting to promote NON_VOTER 85aaf56abb0d4474b9dd4bb4171f557c to VOTER
I20260924 03:22:03.873466 17016 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } }
I20260924 03:22:03.875409 17070 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 1 LEARNER]: Refusing update from remote peer e9fee3161a524093be6e2e6642cb28c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 03:22:03.876096 17013 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 03:22:03.890053 17013 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 85aaf56abb0d4474b9dd4bb4171f557c (127.10.12.189) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } }
I20260924 03:22:03.895124 17016 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e9fee3161a524093be6e2e6642cb28c3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } }
I20260924 03:22:03.895383 17016 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:03.895697 17016 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9fee3161a524093be6e2e6642cb28c3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } }
I20260924 03:22:03.895323 17069 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 85aaf56abb0d4474b9dd4bb4171f557c (127.10.12.189) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } }
I20260924 03:22:03.896492 17016 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:03.900751 17098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9fee3161a524093be6e2e6642cb28c3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } }
I20260924 03:22:03.901033 17098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [sys.catalog]: This master's current role is: FOLLOWER
W20260924 03:22:03.945770 17015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:03.946089 17015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:03.946180 17015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:03.954521 17015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:22:03.954661 17015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:03.954707 17015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:03.954761 17015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:22:03.954815 17015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:22:03.965710 17015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:37301,127.10.12.189:33891,127.10.12.188:36159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.188.keytab
--principal=oryx/127.10.12.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.188:36159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.12.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:03.967170 17015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:03.968639 17015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:03.983850 17116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:03.983829 17117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:03.983872 17119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:03.984550 17015 server_base.cc:1034] running on GCE node
I20260924 03:22:03.985760 17015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:03.987371 17015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:03.988621 17015 hybrid_clock.cc:648] HybridClock initialized: now 1790220123988565 us; error 53 us; skew 500 ppm
I20260924 03:22:03.989068 17015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:03 dist-test-slave-00qs krb5kdc[16408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220123, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:03.992895 17015 init.cc:377] Logged in from keytab as oryx/127.10.12.188@KRBTEST.COM (short username oryx)
I20260924 03:22:03.997349 17015 webserver.cc:460] Webserver started at http://127.10.12.188:35573/ using document root <none> and password file <none>
I20260924 03:22:03.998100 17015 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:03.998227 17015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:03.998533 17015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:04.001802 17015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/data/instance:
uuid: "767765115d1d45b59d593a3a53cc529c"
format_stamp: "Formatted at 2026-09-24 03:22:03 on dist-test-slave-00qs"
server_key: "328ff0e150aa31f4ab5af2e6b5678e89"
server_key_iv: "dd26baa564b3983d1bcfbfa664930b5e"
server_key_version: "encryptionkey@0"
I20260924 03:22:04.002863 17015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/wal/instance:
uuid: "767765115d1d45b59d593a3a53cc529c"
format_stamp: "Formatted at 2026-09-24 03:22:03 on dist-test-slave-00qs"
server_key: "328ff0e150aa31f4ab5af2e6b5678e89"
server_key_iv: "dd26baa564b3983d1bcfbfa664930b5e"
server_key_version: "encryptionkey@0"
I20260924 03:22:04.009640 17015 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260924 03:22:04.014705 17126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:04.016868 17015 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260924 03:22:04.017089 17015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/wal
uuid: "767765115d1d45b59d593a3a53cc529c"
format_stamp: "Formatted at 2026-09-24 03:22:03 on dist-test-slave-00qs"
server_key: "328ff0e150aa31f4ab5af2e6b5678e89"
server_key_iv: "dd26baa564b3983d1bcfbfa664930b5e"
server_key_version: "encryptionkey@0"
I20260924 03:22:04.017309 17015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:04.031798 17015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:04.036700 17015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:04.036965 17015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:04.056908 17015 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.188:36159
I20260924 03:22:04.056931 17177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.188:36159 every 8 connection(s)
I20260924 03:22:04.058359 17015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/data/info.pb
I20260924 03:22:04.060698 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 17015
I20260924 03:22:04.062961 17178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:04.072582 17178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:22:04 dist-test-slave-00qs krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.12.188@KRBTEST.COM
Sep 24 03:22:04 dist-test-slave-00qs krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220123, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.188@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM
I20260924 03:22:04.089112 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.063148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42902 (local address 127.10.12.188:36159)
0924 03:22:04.064221 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:04.064238 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:04.064436 (+ 198us) server_negotiation.cc:407] Connection header received
0924 03:22:04.066115 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:04.066140 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:04.066586 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:04.066951 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:04.069770 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.071273 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.073318 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.074591 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.074758 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.080071 (+ 5313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:04.080082 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:04.080095 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:04.080141 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:04.084182 (+ 4041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.085104 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.085111 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.085119 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.085294 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.085738 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.085742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.085744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.086504 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:04.086774 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:04.087304 (+ 530us) server_negotiation.cc:299] Negotiation successful
0924 03:22:04.087805 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":382,"threads_started":1}
I20260924 03:22:04.092929 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.075703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.0.0.1:35364)
0924 03:22:04.076796 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:04.076851 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:04.077029 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:04.077405 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:04.077412 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:04.078237 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:04.078526 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:04.078537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.080066 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.080071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:04.081919 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:04.081944 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.082998 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.083005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:04.083152 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.083783 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:04.083804 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:04.086520 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:04.090112 (+ 3592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:04.090122 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:04.090129 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:04.090680 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:04.091211 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:04.091214 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:04.091217 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:04.091368 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:04.092017 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:04.092025 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:04.092198 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 03:22:04.092367 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 03:22:04.092636 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":620,"threads_started":1}
I20260924 03:22:04.093036 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.075803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.10.12.190:37301)
0924 03:22:04.076409 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:04.076415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:04.076958 (+ 543us) server_negotiation.cc:407] Connection header received
0924 03:22:04.077147 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:04.077151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:04.077218 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:04.077307 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:04.078747 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.079765 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.082129 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.082753 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.082893 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.086740 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:04.086747 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:04.086752 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:04.086797 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:04.089972 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.090842 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.090847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.090850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.091088 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.091481 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.091485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.091488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.091828 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:04.092189 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:04.092523 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 03:22:04.092732 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":293,"threads_started":1}
I20260924 03:22:04.094847 17178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:22:04 dist-test-slave-00qs krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220123, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.188@KRBTEST.COM for oryx/127.10.12.189@KRBTEST.COM
I20260924 03:22:04.109583 17110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.095928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50000 (local address 127.10.12.189:33891)
0924 03:22:04.096128 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:04.096132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:04.096150 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:22:04.096352 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:04.096356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:04.096425 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:04.096525 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:04.097288 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.098059 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.099327 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.099742 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.099850 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.103240 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:04.103246 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:04.103249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:04.103282 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:04.106394 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.107204 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.107208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.107210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.107353 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.108154 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.108158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.108161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.108484 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:04.108616 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:04.109168 (+ 552us) server_negotiation.cc:299] Negotiation successful
0924 03:22:04.109346 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 03:22:04.109618 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.095736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:33891 (local address 127.0.0.1:50000)
0924 03:22:04.096083 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:04.096101 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:04.096202 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:04.096584 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:04.096588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:04.096987 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:04.097132 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:04.097141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.098210 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.098214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:04.099144 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:04.099157 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.099934 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.099939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:04.100142 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.100701 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:04.100717 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:04.102752 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:04.106595 (+ 3843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:04.106600 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:04.106603 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:04.107069 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:04.107653 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:04.107659 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:04.107662 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:04.107936 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:04.108670 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:04.108675 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:04.108846 (+ 171us) client_negotiation.cc:769] Sending connection context
0924 03:22:04.108985 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 03:22:04.109270 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260924 03:22:04.110807 17178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:22:04 dist-test-slave-00qs krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220123, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.188@KRBTEST.COM for oryx/127.10.12.188@KRBTEST.COM
I20260924 03:22:04.123603 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.111580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:36159 (local address 127.0.0.1:42916)
0924 03:22:04.111789 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:04.111808 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:04.111914 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:04.112240 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:04.112243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:04.112671 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:04.112806 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:04.112815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.113792 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.113796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:04.114741 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:04.114753 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.115519 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.115524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:04.115684 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.116309 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:04.116326 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:04.118340 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:04.121199 (+ 2859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:04.121203 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:04.121206 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:04.121575 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:04.122048 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:04.122051 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:04.122054 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:04.122260 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:04.122811 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:04.122814 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:04.122973 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 03:22:04.123100 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 03:22:04.123330 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 03:22:04.123889 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.111799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42916 (local address 127.10.12.188:36159)
0924 03:22:04.111927 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:04.111931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:04.111947 (+ 16us) server_negotiation.cc:407] Connection header received
0924 03:22:04.112051 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:04.112054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:04.112107 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:04.112180 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:04.112956 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.113630 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.114903 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.115320 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.115425 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.118547 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:04.118552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:04.118556 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:04.118586 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:04.121026 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.121697 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.121701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.121703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.121931 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.122345 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.122349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.122351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.122654 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:04.122763 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:04.123174 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 03:22:04.123512 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":44}
I20260924 03:22:04.129057 17178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c: Bootstrap starting.
I20260924 03:22:04.132781 17178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:04.133836 17178 log.cc:824] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:22:04.136837 17178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c: No bootstrap required, opened a new log
I20260924 03:22:04.144943 17178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } }
I20260924 03:22:04.145299 17178 raft_consensus.cc:374] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:04.145388 17178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 767765115d1d45b59d593a3a53cc529c, State: Initialized, Role: FOLLOWER
I20260924 03:22:04.145974 17178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } }
I20260924 03:22:04.147321 17185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } } }
I20260924 03:22:04.147662 17185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:22:04.147874 17178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [sys.catalog]: configured and running, proceeding with master startup.
W20260924 03:22:04.157717 17196 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:22:04.157891 17196 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:22:04.163115 17015 master_runner.cc:428] Detected that this master 767765115d1d45b59d593a3a53cc529c is joining an existing cluster
I20260924 03:22:04.163293 17015 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.12.188:36159
I20260924 03:22:04.186571 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.167596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.10.12.190:37301)
0924 03:22:04.167817 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:04.167823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:04.168376 (+ 553us) server_negotiation.cc:407] Connection header received
0924 03:22:04.168618 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:04.168624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:04.168730 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:04.168816 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:04.171367 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.172579 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.174763 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.175432 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.175573 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.178570 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:04.178576 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:04.178580 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:04.178617 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:04.182451 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.183330 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.183336 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.183339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.183486 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.184256 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.184261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.184264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.184653 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:04.184805 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:04.185903 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0924 03:22:04.186186 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 03:22:04.186784 17202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.167474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.0.0.1:35368)
0924 03:22:04.168227 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:04.168247 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:04.168441 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:04.169058 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:04.169064 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:04.170669 (+ 1605us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:04.171126 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:04.171143 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.172790 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.172799 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:04.174534 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:04.174552 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.175913 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.175922 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:04.176164 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.176946 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:04.176975 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:04.178283 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:04.182616 (+ 4333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:04.182621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:04.182626 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:04.183188 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:04.183617 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:04.183622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:04.183626 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:04.183827 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:04.184879 (+ 1052us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:04.184885 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:04.185525 (+ 640us) client_negotiation.cc:769] Sending connection context
0924 03:22:04.185928 (+ 403us) client_negotiation.cc:240] Negotiation successful
0924 03:22:04.186260 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":61,"spinlock_wait_cycles":537856,"thread_start_us":353,"threads_started":1}
I20260924 03:22:04.190536 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.168624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:33891 (local address 127.0.0.1:50016)
0924 03:22:04.173013 (+ 4389us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:04.173038 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:04.173249 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:04.173805 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:04.173810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:04.175697 (+ 1887us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:04.176262 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:04.176276 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.177566 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.177570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:04.179354 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:04.179380 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.180794 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.180801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:04.181000 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.181646 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:04.181669 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:04.183008 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:04.187191 (+ 4183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:04.187195 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:04.187199 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:04.187915 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:04.188455 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:04.188460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:04.188464 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:04.188638 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:04.189345 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:04.189349 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:04.189555 (+ 206us) client_negotiation.cc:769] Sending connection context
0924 03:22:04.189694 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 03:22:04.190048 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4222,"thread_start_us":4015,"threads_started":1}
I20260924 03:22:04.190546 17110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.168787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50016 (local address 127.10.12.189:33891)
0924 03:22:04.169006 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:04.169012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:04.173134 (+ 4122us) server_negotiation.cc:407] Connection header received
0924 03:22:04.173404 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:04.173410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:04.173498 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:04.173571 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:04.176484 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.177384 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.179555 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.180258 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.180435 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.183241 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:04.183248 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:04.183253 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:04.183295 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:04.186826 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.188109 (+ 1283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.188115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.188118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.188288 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.188769 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.188774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.188777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.189129 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:04.189270 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:04.189892 (+ 622us) server_negotiation.cc:299] Negotiation successful
0924 03:22:04.190152 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 03:22:04.195789 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 16514
I20260924 03:22:04.196880 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.173198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42922 (local address 127.10.12.188:36159)
0924 03:22:04.173391 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:04.173397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:04.179080 (+ 5683us) server_negotiation.cc:407] Connection header received
0924 03:22:04.179303 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:04.179308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:04.179386 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:04.179462 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:04.180924 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.182039 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.184114 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.184813 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.184913 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.188814 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:04.188824 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:04.188828 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:04.188862 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:04.193633 (+ 4771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.194531 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.194536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.194539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.194688 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.195097 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.195102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.195106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.195550 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:04.195707 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:04.196262 (+ 555us) server_negotiation.cc:299] Negotiation successful
0924 03:22:04.196504 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":62}
I20260924 03:22:04.198448 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.173510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:36159 (local address 127.0.0.1:42922)
0924 03:22:04.178984 (+ 5474us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:04.179006 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:04.179179 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:04.179639 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:04.179643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:04.180393 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:04.180743 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:04.180759 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.182246 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.182251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:04.183968 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:04.183984 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.185234 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.185241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:04.185440 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.186990 (+ 1550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:04.187290 (+ 300us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:04.188599 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:04.193811 (+ 5212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:04.193817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:04.193821 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:04.194390 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:04.194822 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:04.194827 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:04.194831 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:04.194998 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:04.195779 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:04.195783 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:04.195990 (+ 207us) client_negotiation.cc:769] Sending connection context
0924 03:22:04.197622 (+ 1632us) client_negotiation.cc:240] Negotiation successful
0924 03:22:04.197971 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5293,"thread_start_us":3280,"threads_started":1}
I20260924 03:22:04.198747 17185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:22:04.199057 17185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } }
I20260924 03:22:04.200898 17185 leader_election.cc:288] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9fee3161a524093be6e2e6642cb28c3 (127.10.12.190:37301), 85aaf56abb0d4474b9dd4bb4171f557c (127.10.12.189:33891)
I20260924 03:22:04.201815 17069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "767765115d1d45b59d593a3a53cc529c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" is_pre_election: true
I20260924 03:22:04.202977 16463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "767765115d1d45b59d593a3a53cc529c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fee3161a524093be6e2e6642cb28c3" is_pre_election: true
I20260924 03:22:04.204317 17130 leader_election.cc:302] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 767765115d1d45b59d593a3a53cc529c; no voters: 85aaf56abb0d4474b9dd4bb4171f557c, e9fee3161a524093be6e2e6642cb28c3
I20260924 03:22:04.205109 17185 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Sep 24 03:22:04 dist-test-slave-00qs krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220120, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.190@KRBTEST.COM for oryx/127.10.12.188@KRBTEST.COM
Sep 24 03:22:04 dist-test-slave-00qs krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.12.129@KRBTEST.COM
I20260924 03:22:04.214318 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.193756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:36159 (local address 127.0.0.1:42928)
0924 03:22:04.194156 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:04.194175 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:04.194289 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:04.197571 (+ 3282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:04.197576 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:04.198019 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:04.198249 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:04.198262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.200611 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.200616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:04.201885 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:04.201904 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.203061 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.203067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:04.203295 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.204191 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:04.204215 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:04.207190 (+ 2975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:04.211549 (+ 4359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:04.211555 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:04.211559 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:04.212216 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:04.212663 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:04.212668 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:04.212672 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:04.212835 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:04.213361 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:04.213367 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:04.213524 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 03:22:04.213656 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 03:22:04.213979 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235}
I20260924 03:22:04.215287 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.194510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42928 (local address 127.10.12.188:36159)
0924 03:22:04.197122 (+ 2612us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:04.197128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:04.197145 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:22:04.197263 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:04.197270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:04.197378 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:04.197465 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:04.199216 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.200427 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.202251 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.202862 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.207343 (+ 4481us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.207521 (+ 178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:04.207527 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:04.207530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:04.207551 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:04.211308 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.212384 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.212388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.212390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.212536 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.212942 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.212947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.212951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.213233 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:04.213361 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:04.213768 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 03:22:04.213997 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2500,"thread_start_us":3878,"threads_started":1}
I20260924 03:22:04.216879 16453 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.12.188:36159: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "767765115d1d45b59d593a3a53cc529c"
member_type: NON_VOTER
last_known_addr {
host: "127.10.12.188"
port: 36159
}
attrs {
promote: true
}
}
dest_uuid: "e9fee3161a524093be6e2e6642cb28c3"
cas_config_opid_index: 6
I20260924 03:22:04.217757 16453 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: NON_VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: true } }
I20260924 03:22:04.220358 17069 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 1 FOLLOWER]: Refusing update from remote peer e9fee3161a524093be6e2e6642cb28c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 03:22:04.220564 17153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:04.221485 17016 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 03:22:04.222123 17206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.196902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36600 (local address 127.10.12.129:38003)
0924 03:22:04.198374 (+ 1472us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:04.198387 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:04.198440 (+ 53us) server_negotiation.cc:407] Connection header received
0924 03:22:04.198722 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:04.198746 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:04.199085 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:04.201162 (+ 2077us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:04.201502 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.203029 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.205074 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.206028 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.213440 (+ 7412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.213762 (+ 322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:04.213777 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:04.213798 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:04.213833 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:04.217914 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.218747 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.218755 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.218758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.218914 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.219402 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.219406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.219408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.219799 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:04.221218 (+ 1419us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:04.221399 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 03:22:04.221687 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1198,"thread_start_us":1959,"threads_started":1}
I20260924 03:22:04.223562 16637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:04.224377 17153 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 1 FOLLOWER]: Refusing update from remote peer e9fee3161a524093be6e2e6642cb28c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 03:22:04.225238 17016 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 767765115d1d45b59d593a3a53cc529c (127.10.12.188) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: NON_VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: true } } }
I20260924 03:22:04.228138 17014 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: NON_VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:22:04.232299 17069 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 767765115d1d45b59d593a3a53cc529c (127.10.12.188) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: NON_VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: true } } }
I20260924 03:22:04.236227 17016 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.12.188:36159
I20260924 03:22:04.236683 17098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9fee3161a524093be6e2e6642cb28c3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: NON_VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: true } } }
I20260924 03:22:04.237010 17098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:22:04.237808 17015 master.cc:537] Master@127.10.12.188:36159 shutting down...
I20260924 03:22:04.237655 17013 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e9fee3161a524093be6e2e6642cb28c3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: NON_VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: true } } }
I20260924 03:22:04.237950 17013 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:04.239319 17016 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9fee3161a524093be6e2e6642cb28c3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: NON_VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: true } } }
I20260924 03:22:04.239794 17016 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:04.248275 17153 raft_consensus.cc:858] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: NON_VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } }
W20260924 03:22:04.248684 17153 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: NON_VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } }
I20260924 03:22:04.273633 17153 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 85aaf56abb0d4474b9dd4bb4171f557c (127.10.12.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: NON_VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: true } } }
I20260924 03:22:04.278592 17213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9fee3161a524093be6e2e6642cb28c3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: NON_VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: true } } }
I20260924 03:22:04.279006 17213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260924 03:22:04.279495 17185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9fee3161a524093be6e2e6642cb28c3. Latest consensus state: current_term: 1 leader_uuid: "e9fee3161a524093be6e2e6642cb28c3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: NON_VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: true } } }
I20260924 03:22:04.279723 17185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260924 03:22:04.280185 17210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9fee3161a524093be6e2e6642cb28c3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: NON_VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: true } } }
I20260924 03:22:04.280472 17210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260924 03:22:04.281530 17016 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9fee3161a524093be6e2e6642cb28c3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: NON_VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: true } } }
I20260924 03:22:04.281853 17016 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:04.283015 17015 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
W20260924 03:22:04.283171 16440 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 03:22:04.283514 17015 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260924 03:22:04.283676 17015 tablet_replica.cc:330] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c: stopping tablet replica
W20260924 03:22:04.289021 16440 consensus_peers.cc:487] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 -> Peer 767765115d1d45b59d593a3a53cc529c (127.10.12.188:36159): Couldn't send request to peer 767765115d1d45b59d593a3a53cc529c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 03:22:04.292551 17015 master.cc:552] Master@127.10.12.188:36159 shutdown complete.
I20260924 03:22:04.292666 17015 master_runner.cc:305] Clearing existing system tablet
I20260924 03:22:04.293062 17015 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:04.293180 17015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:04.296427 17015 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260924 03:22:04.298292 17216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:04.298727 17015 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:04.299019 17015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260924 03:22:04.299125 17015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/wal
uuid: "767765115d1d45b59d593a3a53cc529c"
format_stamp: "Formatted at 2026-09-24 03:22:03 on dist-test-slave-00qs"
server_key: "328ff0e150aa31f4ab5af2e6b5678e89"
server_key_iv: "dd26baa564b3983d1bcfbfa664930b5e"
server_key_version: "encryptionkey@0"
I20260924 03:22:04.304004 17015 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:22:04.311623 17015 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 03:22:04.311827 17015 log.cc:1196] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 03:22:04.312440 17015 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c: Deleting consensus metadata
I20260924 03:22:04.312696 17015 master_runner.cc:315] Copying system tablet from 127.10.12.190:37301
I20260924 03:22:04.315395 17015 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c: tablet copy: Beginning tablet copy session from remote peer at address 127.10.12.190:37301
I20260924 03:22:04.335065 17222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.316269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.0.0.1:35380)
0924 03:22:04.317102 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:04.317123 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:04.317285 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:04.317869 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:04.317875 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:04.318453 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:04.318737 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:04.318749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.320276 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.320281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:04.321820 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:04.321837 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.323331 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.323338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:04.323539 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.324647 (+ 1108us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:04.324671 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:04.325788 (+ 1117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:04.331638 (+ 5850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:04.331645 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:04.331649 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:04.332261 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:04.332816 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:04.332821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:04.332825 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:04.333108 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:04.333855 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:04.333860 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:04.334049 (+ 189us) client_negotiation.cc:769] Sending connection context
0924 03:22:04.334204 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 03:22:04.334620 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":489,"threads_started":1}
I20260924 03:22:04.335088 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.316352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35380 (local address 127.10.12.190:37301)
0924 03:22:04.316632 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:04.316638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:04.317231 (+ 593us) server_negotiation.cc:407] Connection header received
0924 03:22:04.317429 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:04.317435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:04.317531 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:04.317636 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:04.318919 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.320059 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.322038 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.322756 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.322863 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.326082 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:04.326090 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:04.326095 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:04.326137 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:04.331337 (+ 5200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.332460 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.332466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.332470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.332650 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.333269 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.333275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.333279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.333697 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:04.333880 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:04.334401 (+ 521us) server_negotiation.cc:299] Negotiation successful
0924 03:22:04.334665 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260924 03:22:04.336228 16473 tablet_copy_service.cc:140] P e9fee3161a524093be6e2e6642cb28c3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 767765115d1d45b59d593a3a53cc529c ({username='oryx', principal='oryx/127.10.12.188@KRBTEST.COM'} at 127.0.0.1:35380)
I20260924 03:22:04.336433 16473 tablet_copy_service.cc:161] P e9fee3161a524093be6e2e6642cb28c3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 767765115d1d45b59d593a3a53cc529c at {username='oryx', principal='oryx/127.10.12.188@KRBTEST.COM'} at 127.0.0.1:35380: session id = 767765115d1d45b59d593a3a53cc529c-00000000000000000000000000000000
I20260924 03:22:04.340255 16473 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 03:22:04.342527 17015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 03:22:04.345708 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 16514
I20260924 03:22:04.349609 17015 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c: tablet copy: Starting download of 0 data blocks...
I20260924 03:22:04.349947 17015 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c: tablet copy: Starting download of 1 WAL segments...
I20260924 03:22:04.352660 17015 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 03:22:04.357453 16473 tablet_copy_service.cc:342] P e9fee3161a524093be6e2e6642cb28c3: Request end of tablet copy session 767765115d1d45b59d593a3a53cc529c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.12.188@KRBTEST.COM'} at 127.0.0.1:35380
I20260924 03:22:04.357741 16473 tablet_copy_service.cc:433] P e9fee3161a524093be6e2e6642cb28c3: ending tablet copy session 767765115d1d45b59d593a3a53cc529c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 767765115d1d45b59d593a3a53cc529c
I20260924 03:22:04.362819 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:38003
--local_ip_for_outbound_sockets=127.10.12.129
--tserver_master_addrs=127.10.12.190:37301
--webserver_port=34653
--webserver_interface=127.10.12.129
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.129.keytab
--principal=oryx/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.12.190:37301,127.10.12.189:33891
--tserver_master_addrs=127.10.12.190:37301,127.10.12.189:33891,127.10.12.188:36159 with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:22:04.369875 17015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:04.378024 17224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:04.378459 17225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:04.385911 17015 server_base.cc:1034] running on GCE node
W20260924 03:22:04.386278 17228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:04.386794 17015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:04.387454 17015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:04.388665 17015 hybrid_clock.cc:648] HybridClock initialized: now 1790220124388643 us; error 62 us; skew 500 ppm
I20260924 03:22:04.388926 17015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:04 dist-test-slave-00qs krb5kdc[16408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220124, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:04.391261 17015 init.cc:377] Logged in from keytab as oryx/127.10.12.188@KRBTEST.COM (short username oryx)
I20260924 03:22:04.397121 17015 webserver.cc:460] Webserver started at http://127.10.12.188:33345/ using document root <none> and password file <none>
I20260924 03:22:04.397522 17015 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:04.397619 17015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:04.401237 17015 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:22:04.403412 17234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:04.403882 17015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 03:22:04.404060 17015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/wal
uuid: "767765115d1d45b59d593a3a53cc529c"
format_stamp: "Formatted at 2026-09-24 03:22:03 on dist-test-slave-00qs"
server_key: "328ff0e150aa31f4ab5af2e6b5678e89"
server_key_iv: "dd26baa564b3983d1bcfbfa664930b5e"
server_key_version: "encryptionkey@0"
I20260924 03:22:04.404251 17015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:04.429349 17015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:04.434177 17015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:04.481132 17015 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.188:36159
I20260924 03:22:04.481182 17285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.188:36159 every 8 connection(s)
I20260924 03:22:04.482419 17015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/master-2/data/info.pb
I20260924 03:22:04.488317 17286 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:22:04.492606 17286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c: Bootstrap starting.
I20260924 03:22:04.509472 17286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:22:04.510059 17286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c: Bootstrap complete.
I20260924 03:22:04.511202 17286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: NON_VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: true } }
I20260924 03:22:04.511539 17286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 767765115d1d45b59d593a3a53cc529c, State: Initialized, Role: LEARNER
I20260924 03:22:04.511811 17286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: NON_VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: true } }
I20260924 03:22:04.512720 17289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: NON_VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: true } } }
I20260924 03:22:04.513046 17286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:22:04.513273 17289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [sys.catalog]: This master's current role is: LEARNER
I20260924 03:22:04.524936 17300 catalog_manager.cc:1259] Loaded cluster ID: 9da1f5522cb640e3bc32d53489f76eb7
I20260924 03:22:04.525118 17300 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c: loading cluster ID for follower catalog manager: success
I20260924 03:22:04.528898 17300 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c: acquiring CA information for follower catalog manager: success
I20260924 03:22:04.531524 17300 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 03:22:04.750726 17223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:04.751219 17223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:04.751381 17223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:04.759043 17223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:04.759204 17223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:04.759361 17223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:22:04.767484 17109 catalog_manager.cc:1259] Loaded cluster ID: 9da1f5522cb640e3bc32d53489f76eb7
I20260924 03:22:04.767601 17109 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c: loading cluster ID for follower catalog manager: success
I20260924 03:22:04.770202 17109 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c: acquiring CA information for follower catalog manager: success
I20260924 03:22:04.770527 17223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.129.keytab
--principal=oryx/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:38003
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=34653
--tserver_master_addrs=127.10.12.190:37301,127.10.12.189:33891,127.10.12.188:36159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:04.771986 17223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:04.772295 17109 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:22:04.773360 17223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:04.787410 17305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:04.788025 17306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:04.788298 17308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:04.789885 17223 server_base.cc:1034] running on GCE node
I20260924 03:22:04.790736 17223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:04.791785 17223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:04.793025 17223 hybrid_clock.cc:648] HybridClock initialized: now 1790220124792980 us; error 70 us; skew 500 ppm
I20260924 03:22:04.793383 17223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:04 dist-test-slave-00qs krb5kdc[16408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220124, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:04.796846 17223 init.cc:377] Logged in from keytab as oryx/127.10.12.129@KRBTEST.COM (short username oryx)
I20260924 03:22:04.798784 17223 webserver.cc:460] Webserver started at http://127.10.12.129:34653/ using document root <none> and password file <none>
I20260924 03:22:04.799569 17223 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:04.799729 17223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:04.806630 17223 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:22:04.809798 17316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:04.811702 17223 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260924 03:22:04.811906 17223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "8c17e6da2e044dffbd40657771620fb5"
format_stamp: "Formatted at 2026-09-24 03:22:01 on dist-test-slave-00qs"
server_key: "7fd64b6bd8859b9e4cdb3ac9757fd319"
server_key_iv: "6bd58ac8edf742df9e25019be58eda79"
server_key_version: "encryptionkey@0"
I20260924 03:22:04.813022 17223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:04.822126 17310 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9fee3161a524093be6e2e6642cb28c3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: NON_VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: true } } }
I20260924 03:22:04.822427 17310 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:04.837996 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.822440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:36159 (local address 127.0.0.1:42944)
0924 03:22:04.832220 (+ 9780us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:04.832246 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:04.832438 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:04.833001 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:04.833007 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:04.833033 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:04.833244 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:04.833254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.834546 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.834551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:04.836019 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:04.836033 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.837176 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.837182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:04.837356 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.837419 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 03:22:04.837509 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 03:22:04.837658 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9636,"thread_start_us":9526,"threads_started":1}
I20260924 03:22:04.838009 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.822701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42944 (local address 127.10.12.188:36159)
0924 03:22:04.832610 (+ 9909us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:04.832618 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:04.832636 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:22:04.832714 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:04.832719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:04.832816 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:04.832941 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:04.833395 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.834367 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.836175 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.836996 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.837119 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.837381 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:04.837542 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 03:22:04.837677 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9750,"thread_start_us":9631,"threads_started":1}
I20260924 03:22:04.839018 17261 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260924 03:22:04.839428 17289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9fee3161a524093be6e2e6642cb28c3. Latest consensus state: current_term: 1 leader_uuid: "e9fee3161a524093be6e2e6642cb28c3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: NON_VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: true } } }
I20260924 03:22:04.839708 17289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [sys.catalog]: This master's current role is: LEARNER
I20260924 03:22:04.840129 17310 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3: attempting to promote NON_VOTER 767765115d1d45b59d593a3a53cc529c to VOTER
I20260924 03:22:04.840826 17310 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: false } }
I20260924 03:22:04.842682 17069 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 1 FOLLOWER]: Refusing update from remote peer e9fee3161a524093be6e2e6642cb28c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 03:22:04.843283 17261 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 1 LEARNER]: Refusing update from remote peer e9fee3161a524093be6e2e6642cb28c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 03:22:04.843326 17223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:04.843416 17322 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 03:22:04.843824 17310 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 03:22:04.846318 17322 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 767765115d1d45b59d593a3a53cc529c (127.10.12.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: false } } }
I20260924 03:22:04.847002 17069 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 767765115d1d45b59d593a3a53cc529c (127.10.12.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: false } } }
I20260924 03:22:04.849390 17322 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e9fee3161a524093be6e2e6642cb28c3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: false } } }
I20260924 03:22:04.849627 17322 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:04.849684 17223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:04.849990 17223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:04.849890 17321 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9fee3161a524093be6e2e6642cb28c3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: false } } }
I20260924 03:22:04.850099 17321 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:04.851330 17327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9fee3161a524093be6e2e6642cb28c3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: false } } }
I20260924 03:22:04.851745 17327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:22:04.851814 17310 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9fee3161a524093be6e2e6642cb28c3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: false } } }
I20260924 03:22:04.852162 17310 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:04.852929 17223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:22:04.853276 17261 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 767765115d1d45b59d593a3a53cc529c (127.10.12.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: false } } }
I20260924 03:22:04.854899 17223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:22:04.855566 17223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:04.855702 17223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:22:04.855783 17223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:04.857340 17289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9fee3161a524093be6e2e6642cb28c3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: false } } }
I20260924 03:22:04.857601 17289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:22:04.890134 17223 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:38003
I20260924 03:22:04.890194 17436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:38003 every 8 connection(s)
I20260924 03:22:04.891710 17223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:22:04.893151 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 17223
I20260924 03:22:04.893534 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 16648
Sep 24 03:22:04 dist-test-slave-00qs krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220124, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM
Sep 24 03:22:04 dist-test-slave-00qs krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220124, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.189@KRBTEST.COM
Sep 24 03:22:04 dist-test-slave-00qs krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220124, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.188@KRBTEST.COM
Sep 24 03:22:04 dist-test-slave-00qs krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.12.130@KRBTEST.COM
I20260924 03:22:04.922326 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.895331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:51893 (local address 127.10.12.190:37301)
0924 03:22:04.896165 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:04.896172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:04.896679 (+ 507us) server_negotiation.cc:407] Connection header received
0924 03:22:04.900982 (+ 4303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:04.900987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:04.901072 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:04.901151 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:04.903579 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.904805 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.907656 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.908392 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.908545 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.914960 (+ 6415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:04.914968 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:04.914973 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:04.915016 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:04.919037 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.920118 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.920124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.920128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.920269 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.920697 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.920702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.920705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.921060 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:04.921176 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:04.921716 (+ 540us) server_negotiation.cc:299] Negotiation successful
0924 03:22:04.921951 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":468,"threads_started":1}
I20260924 03:22:04.922763 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.898866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:42753 (local address 127.10.12.188:36159)
0924 03:22:04.899087 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:04.899094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:04.900154 (+ 1060us) server_negotiation.cc:407] Connection header received
0924 03:22:04.900933 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:04.900939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:04.901013 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:04.901106 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:04.903539 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.904659 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.907665 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.908345 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.908471 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.915847 (+ 7376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:04.915854 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:04.915858 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:04.915902 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:04.919249 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.920650 (+ 1401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.920656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.920660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.920844 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.921291 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.921295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.921298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.921617 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:04.921725 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:04.922243 (+ 518us) server_negotiation.cc:299] Negotiation successful
0924 03:22:04.922469 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 03:22:04.923357 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.895532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.129:51893)
0924 03:22:04.896511 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:04.896556 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:04.900467 (+ 3911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:04.901425 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:04.901438 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:04.902205 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:04.903322 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:04.903350 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.905028 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.905034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:04.907470 (+ 2436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:04.907488 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.908901 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.908908 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:04.909185 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.909809 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:04.909831 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:04.914720 (+ 4889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:04.919415 (+ 4695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:04.919419 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:04.919423 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:04.919899 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:04.920403 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:04.920408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:04.920411 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:04.920585 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:04.921256 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:04.921260 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:04.921447 (+ 187us) client_negotiation.cc:769] Sending connection context
0924 03:22:04.922621 (+ 1174us) client_negotiation.cc:240] Negotiation successful
0924 03:22:04.922918 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":154,"spinlock_wait_cycles":16128,"thread_start_us":468,"threads_started":1}
I20260924 03:22:04.923586 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.899061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:36159 (local address 127.10.12.129:42753)
0924 03:22:04.900047 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:04.900067 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:04.900807 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:04.901424 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:04.901438 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:04.902819 (+ 1381us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:04.903318 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:04.903350 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.905068 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.905073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:04.907478 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:04.907488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.908763 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.908769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:04.909489 (+ 720us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.909938 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:04.909955 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:04.915614 (+ 5659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:04.919605 (+ 3991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:04.919610 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:04.919613 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:04.920512 (+ 899us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:04.921001 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:04.921005 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:04.921009 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:04.921176 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:04.921808 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:04.921813 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:04.921966 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 03:22:04.922935 (+ 969us) client_negotiation.cc:240] Negotiation successful
0924 03:22:04.923200 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":649,"thread_start_us":509,"threads_started":1}
I20260924 03:22:04.923996 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.895032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36206 (local address 127.10.12.130:39379)
0924 03:22:04.900176 (+ 5144us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:04.900183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:04.900207 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:22:04.900266 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:04.900273 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:04.900495 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:04.900639 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:04.902833 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.905064 (+ 2231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.907055 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.907975 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.909017 (+ 1042us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.915264 (+ 6247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:04.915284 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:04.915302 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:04.915348 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:04.919455 (+ 4107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.920652 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.920661 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.920665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.920824 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.921276 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.921281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.921285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.921684 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:04.923148 (+ 1464us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:04.923311 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 03:22:04.923582 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4997,"thread_start_us":4838,"threads_started":1}
I20260924 03:22:04.924786 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.897306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:35527 (local address 127.10.12.189:33891)
0924 03:22:04.899790 (+ 2484us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:04.899797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:04.899822 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:22:04.900902 (+ 1080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:04.900908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:04.900985 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:04.901633 (+ 648us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:04.903578 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.904805 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.907683 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.908391 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.909259 (+ 868us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.913885 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:04.913890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:04.913894 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:04.913928 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:04.917261 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.918339 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.918344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.918348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.918495 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:04.919028 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:04.919033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:04.919035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:04.919370 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:04.919503 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:04.923257 (+ 3754us) server_negotiation.cc:299] Negotiation successful
0924 03:22:04.923917 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2350,"thread_start_us":2210,"threads_started":1}
I20260924 03:22:04.925117 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:04.897206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:33891 (local address 127.10.12.129:35527)
0924 03:22:04.898504 (+ 1298us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:04.898526 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:04.900479 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:04.901425 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:04.901438 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:04.902234 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:04.903319 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:04.903350 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.905029 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.905034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:04.907469 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:04.907488 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:04.908642 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:04.908651 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:04.909028 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:04.909787 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:04.909818 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:04.913627 (+ 3809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:04.917463 (+ 3836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:04.917476 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:04.917498 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:04.918173 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:04.918636 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:04.918641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:04.918645 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:04.918894 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:04.919514 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:04.919522 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:04.919892 (+ 370us) client_negotiation.cc:769] Sending connection context
0924 03:22:04.920265 (+ 373us) client_negotiation.cc:240] Negotiation successful
0924 03:22:04.920825 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1136,"mutex_wait_us":398,"spinlock_wait_cycles":1920,"thread_start_us":923,"threads_started":1}
I20260924 03:22:04.925256 17438 heartbeater.cc:347] Connected to a master server at 127.10.12.190:37301
I20260924 03:22:04.925550 16771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:04.925580 17437 heartbeater.cc:347] Connected to a master server at 127.10.12.188:36159
I20260924 03:22:04.925724 17438 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:04.925726 17437 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:04.926759 17439 heartbeater.cc:347] Connected to a master server at 127.10.12.189:33891
I20260924 03:22:04.926931 17439 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:04.927002 17438 heartbeater.cc:511] Master 127.10.12.190:37301 requested a full tablet report, sending...
I20260924 03:22:04.927034 17437 heartbeater.cc:511] Master 127.10.12.188:36159 requested a full tablet report, sending...
I20260924 03:22:04.927366 17439 heartbeater.cc:511] Master 127.10.12.189:33891 requested a full tablet report, sending...
I20260924 03:22:04.929232 16453 ts_manager.cc:194] Re-registered known tserver with Master: 8c17e6da2e044dffbd40657771620fb5 (127.10.12.129:38003)
I20260924 03:22:04.929416 17060 ts_manager.cc:194] Registered new tserver with Master: 8c17e6da2e044dffbd40657771620fb5 (127.10.12.129:38003)
I20260924 03:22:04.929504 17251 ts_manager.cc:194] Registered new tserver with Master: 8c17e6da2e044dffbd40657771620fb5 (127.10.12.129:38003)
I20260924 03:22:04.930305 16453 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.12.129@KRBTEST.COM'} at 127.10.12.129:51893
I20260924 03:22:05.019799 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 16648
I20260924 03:22:05.033378 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:39379
--local_ip_for_outbound_sockets=127.10.12.130
--tserver_master_addrs=127.10.12.190:37301
--webserver_port=36451
--webserver_interface=127.10.12.130
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.130.keytab
--principal=oryx/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.12.190:37301,127.10.12.189:33891
--tserver_master_addrs=127.10.12.190:37301,127.10.12.189:33891,127.10.12.188:36159 with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:22:05.369123 17449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:05.369421 17449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:05.369513 17449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:05.377049 17449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:05.377182 17449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:05.377295 17449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:22:05.387408 17449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.130.keytab
--principal=oryx/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:39379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=36451
--tserver_master_addrs=127.10.12.190:37301,127.10.12.189:33891,127.10.12.188:36159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:05.388760 17449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:05.390049 17449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:05.404057 17455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:05.403985 17454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:05.404017 17457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:05.404799 17449 server_base.cc:1034] running on GCE node
I20260924 03:22:05.405531 17449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:05.406665 17449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:05.407910 17449 hybrid_clock.cc:648] HybridClock initialized: now 1790220125407848 us; error 54 us; skew 500 ppm
I20260924 03:22:05.408270 17449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:05 dist-test-slave-00qs krb5kdc[16408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220125, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:05.412036 17449 init.cc:377] Logged in from keytab as oryx/127.10.12.130@KRBTEST.COM (short username oryx)
I20260924 03:22:05.414064 17449 webserver.cc:460] Webserver started at http://127.10.12.130:36451/ using document root <none> and password file <none>
I20260924 03:22:05.415107 17449 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:05.415284 17449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:05.421945 17449 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 03:22:05.425114 17464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:05.427040 17449 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 03:22:05.427217 17449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "3cb91631ae5147c6aca234d1afe3b4cc"
format_stamp: "Formatted at 2026-09-24 03:22:02 on dist-test-slave-00qs"
server_key: "84cc17252850f93a7707f10f96dd1a94"
server_key_iv: "cc8c9d11b4fafa8b31e1eb5241cb27fc"
server_key_version: "encryptionkey@0"
I20260924 03:22:05.428274 17449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:05.447134 17449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:05.452322 17449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:05.452590 17449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:05.453758 17449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:22:05.455286 17449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:22:05.455384 17449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:05.455464 17449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:22:05.455518 17449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:05.488946 17449 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:39379
I20260924 03:22:05.489028 17576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:39379 every 8 connection(s)
I20260924 03:22:05.490383 17449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:22:05.495347 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 17449
I20260924 03:22:05.495641 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 16782
Sep 24 03:22:05 dist-test-slave-00qs krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220125, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.188@KRBTEST.COM
Sep 24 03:22:05 dist-test-slave-00qs krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220125, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM
Sep 24 03:22:05 dist-test-slave-00qs krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.12.131@KRBTEST.COM
Sep 24 03:22:05 dist-test-slave-00qs krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220125, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.189@KRBTEST.COM
I20260924 03:22:05.518287 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:05.493296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:37269 (local address 127.10.12.188:36159)
0924 03:22:05.493794 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:05.493798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:05.495060 (+ 1262us) server_negotiation.cc:407] Connection header received
0924 03:22:05.497811 (+ 2751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:05.497817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:05.497902 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:05.497980 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:05.500877 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:05.502015 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:05.504879 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:05.505487 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:05.505593 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:05.510452 (+ 4859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:05.510459 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:05.510463 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:05.510513 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:05.514332 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:05.515494 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:05.515512 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:05.515516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:05.515665 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:05.516220 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:05.516225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:05.516228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:05.516585 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:05.516707 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:05.517690 (+ 983us) server_negotiation.cc:299] Negotiation successful
0924 03:22:05.517915 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":265,"threads_started":1}
I20260924 03:22:05.519172 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:05.495647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:48741 (local address 127.10.12.190:37301)
0924 03:22:05.496375 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:05.496381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:05.496888 (+ 507us) server_negotiation.cc:407] Connection header received
0924 03:22:05.497886 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:05.497891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:05.497970 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:05.498072 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:05.500876 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:05.501994 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:05.504913 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:05.505524 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:05.505616 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:05.510507 (+ 4891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:05.510514 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:05.510518 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:05.510560 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:05.514963 (+ 4403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:05.515836 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:05.515842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:05.515845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:05.516063 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:05.516468 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:05.516472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:05.516475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:05.516881 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:05.517086 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:05.518488 (+ 1402us) server_negotiation.cc:299] Negotiation successful
0924 03:22:05.518816 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":497,"threads_started":1}
I20260924 03:22:05.520762 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:05.495483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.130:48741)
0924 03:22:05.496785 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:05.496805 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:05.497471 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:05.498456 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:05.498469 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:05.499446 (+ 977us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:05.500652 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:05.500682 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:05.502219 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:05.502224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:05.504730 (+ 2506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:05.504747 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:05.505967 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:05.505974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:05.506555 (+ 581us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:05.507294 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:05.507315 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:05.510263 (+ 2948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:05.515141 (+ 4878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:05.515146 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:05.515150 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:05.515711 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:05.516195 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:05.516199 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:05.516202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:05.516345 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:05.517145 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:05.517149 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:05.517333 (+ 184us) client_negotiation.cc:769] Sending connection context
0924 03:22:05.517576 (+ 243us) client_negotiation.cc:240] Negotiation successful
0924 03:22:05.518172 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1171,"mutex_wait_us":725,"spinlock_wait_cycles":18176,"thread_start_us":1082,"threads_started":1}
I20260924 03:22:05.520780 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:05.494013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:36159 (local address 127.10.12.130:37269)
0924 03:22:05.494859 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:05.494919 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:05.497471 (+ 2552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:05.498456 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:05.498469 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:05.499446 (+ 977us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:05.500652 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:05.500682 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:05.502219 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:05.502224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:05.504715 (+ 2491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:05.504728 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:05.505943 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:05.505951 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:05.506306 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:05.506955 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:05.506987 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:05.510158 (+ 3171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:05.514602 (+ 4444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:05.514616 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:05.514644 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:05.515261 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:05.515807 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:05.515811 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:05.515814 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:05.516104 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:05.516803 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:05.516812 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:05.517264 (+ 452us) client_negotiation.cc:769] Sending connection context
0924 03:22:05.518761 (+ 1497us) client_negotiation.cc:240] Negotiation successful
0924 03:22:05.519055 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":265,"spinlock_wait_cycles":19200,"thread_start_us":360,"threads_started":1}
I20260924 03:22:05.521951 17586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:05.496704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.10.12.131:46457)
0924 03:22:05.497425 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:05.497433 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:05.498322 (+ 889us) server_negotiation.cc:407] Connection header received
0924 03:22:05.498406 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:05.498415 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:05.498707 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:05.498858 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:05.501240 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:05.502886 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:05.504309 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:05.505206 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:05.505321 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:05.512045 (+ 6724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:05.512067 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:05.512088 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:05.512140 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:05.516373 (+ 4233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:05.518551 (+ 2178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:05.518563 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:05.518566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:05.518764 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:05.519266 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:05.519272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:05.519275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:05.519753 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:05.520066 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:05.521259 (+ 1193us) server_negotiation.cc:299] Negotiation successful
0924 03:22:05.521561 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":412,"threads_started":1}
I20260924 03:22:05.522923 17588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:05.497863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:33891 (local address 127.10.12.130:34717)
0924 03:22:05.504309 (+ 6446us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:05.504332 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:05.504535 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:05.505149 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:05.505153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:05.505714 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:05.506142 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:05.506154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:05.507564 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:05.507569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:05.510880 (+ 3311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:05.510896 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:05.512109 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:05.512115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:05.512284 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:05.512901 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:05.512924 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:05.516069 (+ 3145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:05.520016 (+ 3947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:05.520022 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:05.520026 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:05.520528 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:05.520953 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:05.520962 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:05.520965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:05.521255 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:05.521933 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:05.521938 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:05.522099 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 03:22:05.522251 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 03:22:05.522527 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6304,"thread_start_us":4993,"threads_started":1}
I20260924 03:22:05.522902 17578 heartbeater.cc:347] Connected to a master server at 127.10.12.190:37301
I20260924 03:22:05.523228 17578 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:05.523319 16905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:05.522974 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:05.497172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:34717 (local address 127.10.12.189:33891)
0924 03:22:05.497627 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:05.497634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:05.504420 (+ 6786us) server_negotiation.cc:407] Connection header received
0924 03:22:05.504704 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:05.504710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:05.504855 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:05.504934 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:05.506274 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:05.507392 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:05.511085 (+ 3693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:05.511692 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:05.511791 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:05.516375 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:05.516382 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:05.516386 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:05.516429 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:05.519832 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:05.520655 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:05.520660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:05.520663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:05.520808 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:05.521378 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:05.521384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:05.521387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:05.521731 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:05.521852 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:05.522393 (+ 541us) server_negotiation.cc:299] Negotiation successful
0924 03:22:05.522625 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":226,"threads_started":1}
I20260924 03:22:05.524061 17577 heartbeater.cc:347] Connected to a master server at 127.10.12.188:36159
I20260924 03:22:05.524225 17577 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:05.524726 17578 heartbeater.cc:511] Master 127.10.12.190:37301 requested a full tablet report, sending...
I20260924 03:22:05.524920 17579 heartbeater.cc:347] Connected to a master server at 127.10.12.189:33891
I20260924 03:22:05.525041 17579 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:05.525414 17579 heartbeater.cc:511] Master 127.10.12.189:33891 requested a full tablet report, sending...
I20260924 03:22:05.525899 17577 heartbeater.cc:511] Master 127.10.12.188:36159 requested a full tablet report, sending...
I20260924 03:22:05.526614 16453 ts_manager.cc:194] Re-registered known tserver with Master: 3cb91631ae5147c6aca234d1afe3b4cc (127.10.12.130:39379)
I20260924 03:22:05.526729 17060 ts_manager.cc:194] Registered new tserver with Master: 3cb91631ae5147c6aca234d1afe3b4cc (127.10.12.130:39379)
I20260924 03:22:05.527710 16453 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.12.130@KRBTEST.COM'} at 127.10.12.130:48741
I20260924 03:22:05.527938 17251 ts_manager.cc:194] Registered new tserver with Master: 3cb91631ae5147c6aca234d1afe3b4cc (127.10.12.130:39379)
I20260924 03:22:05.609045 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 16782
I20260924 03:22:05.621790 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:46457
--local_ip_for_outbound_sockets=127.10.12.131
--tserver_master_addrs=127.10.12.190:37301
--webserver_port=44201
--webserver_interface=127.10.12.131
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.131.keytab
--principal=oryx/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.12.190:37301,127.10.12.189:33891
--tserver_master_addrs=127.10.12.190:37301,127.10.12.189:33891,127.10.12.188:36159 with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:22:05.933312 17438 heartbeater.cc:503] Master 127.10.12.190:37301 was elected leader, sending a full tablet report...
W20260924 03:22:05.961084 17590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:05.961400 17590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:05.961494 17590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:05.969347 17590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:05.969493 17590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:05.969615 17590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:22:05.981150 17590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.131.keytab
--principal=oryx/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:46457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=44201
--tserver_master_addrs=127.10.12.190:37301,127.10.12.189:33891,127.10.12.188:36159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:05.982846 17590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:05.984602 17590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:06.000950 17596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:06.001062 17595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:06.001060 17598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:06.002347 17590 server_base.cc:1034] running on GCE node
I20260924 03:22:06.003089 17590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:06.004228 17590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:06.005488 17590 hybrid_clock.cc:648] HybridClock initialized: now 1790220126005407 us; error 73 us; skew 500 ppm
I20260924 03:22:06.005926 17590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:06 dist-test-slave-00qs krb5kdc[16408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220126, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:06.009785 17590 init.cc:377] Logged in from keytab as oryx/127.10.12.131@KRBTEST.COM (short username oryx)
I20260924 03:22:06.011788 17590 webserver.cc:460] Webserver started at http://127.10.12.131:44201/ using document root <none> and password file <none>
I20260924 03:22:06.012583 17590 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:06.012737 17590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:06.019150 17590 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 03:22:06.022509 17605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:06.024243 17590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 03:22:06.024459 17590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "82c9a19bc3f44bf0bf0739243ee16066"
format_stamp: "Formatted at 2026-09-24 03:22:02 on dist-test-slave-00qs"
server_key: "ea30e4073bbaf509c08c8d5c2583ccbf"
server_key_iv: "08718d0528896517a96030f6902fdd27"
server_key_version: "encryptionkey@0"
I20260924 03:22:06.025537 17590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:06.052495 17590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:06.057099 17590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:06.057401 17590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:06.058475 17590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:22:06.060135 17590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:22:06.060235 17590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:06.060336 17590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:22:06.060411 17590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:06.091730 17590 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:46457
I20260924 03:22:06.091792 17717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:46457 every 8 connection(s)
I20260924 03:22:06.093204 17590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:22:06.096777 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 17590
Sep 24 03:22:06 dist-test-slave-00qs krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220126, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM
Sep 24 03:22:06 dist-test-slave-00qs krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220126, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.188@KRBTEST.COM
I20260924 03:22:06.127554 17723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:06.096694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.131:42335)
0924 03:22:06.097575 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:06.097635 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:06.100517 (+ 2882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:06.103221 (+ 2704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:06.103241 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:06.104296 (+ 1055us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:06.105520 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:06.105554 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.108030 (+ 2476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.108038 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:06.110535 (+ 2497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:06.110562 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.111904 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.111913 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:06.112322 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:06.113356 (+ 1034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:06.113391 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:06.117359 (+ 3968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:06.122110 (+ 4751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:06.122123 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:06.122155 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:06.122949 (+ 794us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:06.123460 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:06.123465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:06.123470 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:06.123791 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:06.124909 (+ 1118us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:06.124918 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:06.125347 (+ 429us) client_negotiation.cc:769] Sending connection context
0924 03:22:06.125675 (+ 328us) client_negotiation.cc:240] Negotiation successful
0924 03:22:06.126147 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":327,"threads_started":1}
I20260924 03:22:06.134749 17722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:06.096126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:42335 (local address 127.10.12.190:37301)
0924 03:22:06.096615 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:06.096622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:06.097797 (+ 1175us) server_negotiation.cc:407] Connection header received
0924 03:22:06.102437 (+ 4640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:06.102446 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:06.102574 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:06.105808 (+ 3234us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:06.106477 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.107733 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.110889 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.111584 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.112885 (+ 1301us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:06.117729 (+ 4844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:06.117738 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:06.117744 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:06.117797 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:06.121866 (+ 4069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:06.123121 (+ 1255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:06.123128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:06.123132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:06.123304 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:06.123933 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:06.123939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:06.123943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:06.124667 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:06.124825 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:06.128063 (+ 3238us) server_negotiation.cc:299] Negotiation successful
0924 03:22:06.128328 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":259,"threads_started":1}
I20260924 03:22:06.135131 17725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:06.098167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:36159 (local address 127.10.12.131:35371)
0924 03:22:06.116772 (+ 18605us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:06.116795 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:06.117092 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:06.117817 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:06.117821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:06.118550 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:06.118928 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:06.118945 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.120664 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.120671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:06.122374 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:06.122393 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.123653 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.123660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:06.123872 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:06.124538 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:06.124563 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:06.127509 (+ 2946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:06.131925 (+ 4416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:06.131933 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:06.131938 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:06.132670 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:06.133168 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:06.133174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:06.133178 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:06.133354 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:06.134084 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:06.134090 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:06.134271 (+ 181us) client_negotiation.cc:769] Sending connection context
0924 03:22:06.134414 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 03:22:06.134726 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18453,"thread_start_us":17661,"threads_started":1}
I20260924 03:22:06.136430 17726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:06.098289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:35371 (local address 127.10.12.188:36159)
0924 03:22:06.115659 (+ 17370us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:06.115668 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:06.116950 (+ 1282us) server_negotiation.cc:407] Connection header received
0924 03:22:06.117389 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:06.117396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:06.117493 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:06.117595 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:06.119164 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.120416 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.122577 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.123246 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.123350 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:06.127784 (+ 4434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:06.127791 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:06.127796 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:06.127839 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:06.131665 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:06.132831 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:06.132839 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:06.132844 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:06.133024 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:06.133492 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:06.133497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:06.133501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:06.133852 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:06.134018 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:06.135424 (+ 1406us) server_negotiation.cc:299] Negotiation successful
0924 03:22:06.135699 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17199,"thread_start_us":17785,"threads_started":1}
I20260924 03:22:06.140058 17718 heartbeater.cc:347] Connected to a master server at 127.10.12.188:36159
I20260924 03:22:06.140403 17718 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:06.141549 17718 heartbeater.cc:511] Master 127.10.12.188:36159 requested a full tablet report, sending...
I20260924 03:22:06.144024 17251 ts_manager.cc:194] Registered new tserver with Master: 82c9a19bc3f44bf0bf0739243ee16066 (127.10.12.131:46457)
Sep 24 03:22:06 dist-test-slave-00qs krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220126, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.189@KRBTEST.COM
I20260924 03:22:06.146416 17719 heartbeater.cc:347] Connected to a master server at 127.10.12.190:37301
I20260924 03:22:06.146567 17719 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:06.147071 17719 heartbeater.cc:511] Master 127.10.12.190:37301 requested a full tablet report, sending...
I20260924 03:22:06.154331 16453 ts_manager.cc:194] Re-registered known tserver with Master: 82c9a19bc3f44bf0bf0739243ee16066 (127.10.12.131:46457)
I20260924 03:22:06.155665 16453 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.12.131@KRBTEST.COM'} at 127.10.12.131:42335
I20260924 03:22:06.159220 17731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:06.122618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:33891 (local address 127.10.12.131:42481)
0924 03:22:06.132276 (+ 9658us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:06.132303 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:06.132471 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:06.135235 (+ 2764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:06.135241 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:06.135895 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:06.136240 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:06.136254 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.138006 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.138014 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:06.139783 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:06.139803 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.141233 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.141241 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:06.141474 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:06.142238 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:06.142266 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:06.145553 (+ 3287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:06.152550 (+ 6997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:06.152556 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:06.152561 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:06.153219 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:06.156455 (+ 3236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:06.156462 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:06.156467 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:06.157343 (+ 876us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:06.158152 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:06.158156 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:06.158312 (+ 156us) client_negotiation.cc:769] Sending connection context
0924 03:22:06.158465 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 03:22:06.158798 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9473,"thread_start_us":9934,"threads_started":1}
I20260924 03:22:06.160199 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:06.116550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:42481 (local address 127.10.12.189:33891)
0924 03:22:06.134684 (+ 18134us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:06.134692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:06.134714 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:22:06.134782 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:06.134787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:06.134873 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:06.134991 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:06.136548 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.137826 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.140054 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.140723 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.140841 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:06.146192 (+ 5351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:06.146199 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:06.146204 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:06.146272 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:06.150554 (+ 4282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:06.153416 (+ 2862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:06.153424 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:06.153428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:06.153631 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:06.157515 (+ 3884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:06.157522 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:06.157527 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:06.157912 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:06.158060 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:06.159624 (+ 1564us) server_negotiation.cc:299] Negotiation successful
0924 03:22:06.159840 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17972,"thread_start_us":19823,"threads_started":1}
I20260924 03:22:06.161123 17720 heartbeater.cc:347] Connected to a master server at 127.10.12.189:33891
I20260924 03:22:06.161288 17720 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:06.161747 17720 heartbeater.cc:511] Master 127.10.12.189:33891 requested a full tablet report, sending...
I20260924 03:22:06.163015 17060 ts_manager.cc:194] Registered new tserver with Master: 82c9a19bc3f44bf0bf0739243ee16066 (127.10.12.131:46457)
I20260924 03:22:06.167922 17722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:06.128488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35386 (local address 127.10.12.190:37301)
0924 03:22:06.136038 (+ 7550us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:06.136044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:06.146200 (+ 10156us) server_negotiation.cc:407] Connection header received
0924 03:22:06.147863 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:06.147869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:06.148023 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:06.148130 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:06.150578 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.152524 (+ 1946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.154811 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.155591 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.155735 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:06.160657 (+ 4922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:06.160666 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:06.160672 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:06.160723 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:06.164421 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:06.165202 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:06.165208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:06.165211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:06.165381 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:06.165844 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:06.165849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:06.165853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:06.166599 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:06.166854 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:06.167298 (+ 444us) server_negotiation.cc:299] Negotiation successful
0924 03:22:06.167565 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7382,"thread_start_us":16966,"threads_started":1}
I20260924 03:22:06.179085 17735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:06.146366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50028 (local address 127.10.12.189:33891)
0924 03:22:06.157024 (+ 10658us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:06.157030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:06.160913 (+ 3883us) server_negotiation.cc:407] Connection header received
0924 03:22:06.160964 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:06.160969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:06.161040 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:06.161136 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:06.163566 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.164679 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.166818 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.167345 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.167480 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:06.171189 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:06.171195 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:06.171198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:06.171233 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:06.174760 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:06.175597 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:06.175603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:06.175607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:06.175764 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:06.177333 (+ 1569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:06.177338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:06.177342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:06.177760 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:06.178011 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:06.178410 (+ 399us) server_negotiation.cc:299] Negotiation successful
0924 03:22:06.178672 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10512,"thread_start_us":10329,"threads_started":1}
I20260924 03:22:06.180439 17726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:06.148450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42956 (local address 127.10.12.188:36159)
0924 03:22:06.148701 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:06.148708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:06.164229 (+ 15521us) server_negotiation.cc:407] Connection header received
0924 03:22:06.164340 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:06.164346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:06.164430 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:06.164535 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:06.167335 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.168401 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.170570 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.171198 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.171323 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:06.174365 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:06.174373 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:06.174378 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:06.174424 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:06.177167 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:06.177968 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:06.177973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:06.177977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:06.178127 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:06.178548 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:06.178553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:06.178556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:06.178916 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:06.179153 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:06.179830 (+ 677us) server_negotiation.cc:299] Negotiation successful
0924 03:22:06.180112 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260924 03:22:06.172863 16453 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:22:06.182250 16453 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:22:06.241150 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:06.210095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54616 (local address 127.10.12.131:46457)
0924 03:22:06.225718 (+ 15623us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:06.225730 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:06.231307 (+ 5577us) server_negotiation.cc:407] Connection header received
0924 03:22:06.231445 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:06.231455 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:06.231795 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:06.232061 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:06.232953 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.234758 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.237469 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.239083 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.239226 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:06.239462 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:06.240444 (+ 982us) server_negotiation.cc:299] Negotiation successful
0924 03:22:06.240642 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15388,"thread_start_us":15651,"threads_started":1}
I20260924 03:22:06.241142 17750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:06.209096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:46457 (local address 127.0.0.1:54616)
0924 03:22:06.231051 (+ 21955us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:06.231069 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:06.231186 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:06.232360 (+ 1174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:06.232365 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:06.232392 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:06.232738 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:06.232748 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.234901 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.234907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:06.237305 (+ 2398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:06.237321 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.240087 (+ 2766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.240092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:06.240240 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:06.240268 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 03:22:06.240364 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 03:22:06.240663 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21844,"mutex_wait_us":51,"thread_start_us":21713,"threads_started":1}
I20260924 03:22:06.241770 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:06.208699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:38003 (local address 127.0.0.1:36602)
0924 03:22:06.230242 (+ 21543us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:06.230268 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:06.230522 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:06.234039 (+ 3517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:06.234050 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:06.234088 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:06.234869 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:06.234884 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.236864 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.236870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:06.238752 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:06.238768 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.240832 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.240839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:06.241026 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:06.241061 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 03:22:06.241146 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 03:22:06.241310 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21275,"mutex_wait_us":3,"thread_start_us":25708,"threads_started":1}
I20260924 03:22:06.242861 17749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:06.208984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36602 (local address 127.10.12.129:38003)
0924 03:22:06.233290 (+ 24306us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:06.233299 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:06.233340 (+ 41us) server_negotiation.cc:407] Connection header received
0924 03:22:06.233467 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:06.233474 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:06.233744 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:06.233994 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:06.235109 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.236698 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.238975 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.240573 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.242023 (+ 1450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:06.242257 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:06.242346 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 03:22:06.242549 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24114,"thread_start_us":24008,"threads_started":1}
I20260924 03:22:06.244674 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:06.210680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36212 (local address 127.10.12.130:39379)
0924 03:22:06.229619 (+ 18939us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:06.229629 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:06.234916 (+ 5287us) server_negotiation.cc:407] Connection header received
0924 03:22:06.236335 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:06.236343 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:06.236600 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:06.236790 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:06.237689 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.239802 (+ 2113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.242462 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.243565 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.243799 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:06.244064 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:06.244206 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 03:22:06.244375 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18709,"thread_start_us":18537,"threads_started":1}
I20260924 03:22:06.244701 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:06.210163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:39379 (local address 127.0.0.1:36212)
0924 03:22:06.234728 (+ 24565us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:06.234749 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:06.236123 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:06.236798 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:06.236803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:06.236829 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:06.237473 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:06.237488 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.240010 (+ 2522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.240015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:06.242278 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:06.242295 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.243794 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.243801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:06.244038 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:06.244073 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 03:22:06.244167 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 03:22:06.244307 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24428,"mutex_wait_us":44,"thread_start_us":19657,"threads_started":1}
I20260924 03:22:06.246114 17653 tablet_service.cc:1467] Processing CreateTablet for tablet ede60edd66d942dd88f89179c7bc6111 (DEFAULT_TABLE table=test-table [id=ad2c24af36ff4e8181b199609fab677d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:22:06.246613 17372 tablet_service.cc:1467] Processing CreateTablet for tablet ede60edd66d942dd88f89179c7bc6111 (DEFAULT_TABLE table=test-table [id=ad2c24af36ff4e8181b199609fab677d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:22:06.247943 17372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede60edd66d942dd88f89179c7bc6111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:06.247763 17653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede60edd66d942dd88f89179c7bc6111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:06.248029 17512 tablet_service.cc:1467] Processing CreateTablet for tablet ede60edd66d942dd88f89179c7bc6111 (DEFAULT_TABLE table=test-table [id=ad2c24af36ff4e8181b199609fab677d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:22:06.249191 17512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede60edd66d942dd88f89179c7bc6111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:06.262838 17754 tablet_bootstrap.cc:492] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc: Bootstrap starting.
I20260924 03:22:06.266058 17755 tablet_bootstrap.cc:492] T ede60edd66d942dd88f89179c7bc6111 P 8c17e6da2e044dffbd40657771620fb5: Bootstrap starting.
I20260924 03:22:06.267455 17754 tablet_bootstrap.cc:654] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:06.269307 17756 tablet_bootstrap.cc:492] T ede60edd66d942dd88f89179c7bc6111 P 82c9a19bc3f44bf0bf0739243ee16066: Bootstrap starting.
I20260924 03:22:06.270485 17754 log.cc:824] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc: Log is configured to *not* fsync() on all Append() calls
I20260924 03:22:06.272048 17755 tablet_bootstrap.cc:654] T ede60edd66d942dd88f89179c7bc6111 P 8c17e6da2e044dffbd40657771620fb5: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:06.273398 17755 log.cc:824] T ede60edd66d942dd88f89179c7bc6111 P 8c17e6da2e044dffbd40657771620fb5: Log is configured to *not* fsync() on all Append() calls
I20260924 03:22:06.273778 17756 tablet_bootstrap.cc:654] T ede60edd66d942dd88f89179c7bc6111 P 82c9a19bc3f44bf0bf0739243ee16066: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:06.274709 17754 tablet_bootstrap.cc:492] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc: No bootstrap required, opened a new log
I20260924 03:22:06.274772 17756 log.cc:824] T ede60edd66d942dd88f89179c7bc6111 P 82c9a19bc3f44bf0bf0739243ee16066: Log is configured to *not* fsync() on all Append() calls
I20260924 03:22:06.275074 17754 ts_tablet_manager.cc:1397] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260924 03:22:06.277547 17756 tablet_bootstrap.cc:492] T ede60edd66d942dd88f89179c7bc6111 P 82c9a19bc3f44bf0bf0739243ee16066: No bootstrap required, opened a new log
I20260924 03:22:06.277828 17756 ts_tablet_manager.cc:1397] T ede60edd66d942dd88f89179c7bc6111 P 82c9a19bc3f44bf0bf0739243ee16066: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 03:22:06.286499 17755 tablet_bootstrap.cc:492] T ede60edd66d942dd88f89179c7bc6111 P 8c17e6da2e044dffbd40657771620fb5: No bootstrap required, opened a new log
I20260924 03:22:06.286860 17755 ts_tablet_manager.cc:1397] T ede60edd66d942dd88f89179c7bc6111 P 8c17e6da2e044dffbd40657771620fb5: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.011s
I20260924 03:22:06.286942 17754 raft_consensus.cc:348] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c17e6da2e044dffbd40657771620fb5" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38003 } } peers { permanent_uuid: "82c9a19bc3f44bf0bf0739243ee16066" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46457 } } peers { permanent_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 39379 } }
I20260924 03:22:06.287374 17754 raft_consensus.cc:374] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:06.287477 17754 raft_consensus.cc:729] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb91631ae5147c6aca234d1afe3b4cc, State: Initialized, Role: FOLLOWER
I20260924 03:22:06.288163 17754 consensus_queue.cc:260] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c17e6da2e044dffbd40657771620fb5" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38003 } } peers { permanent_uuid: "82c9a19bc3f44bf0bf0739243ee16066" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46457 } } peers { permanent_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 39379 } }
I20260924 03:22:06.288502 17756 raft_consensus.cc:348] T ede60edd66d942dd88f89179c7bc6111 P 82c9a19bc3f44bf0bf0739243ee16066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c17e6da2e044dffbd40657771620fb5" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38003 } } peers { permanent_uuid: "82c9a19bc3f44bf0bf0739243ee16066" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46457 } } peers { permanent_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 39379 } }
I20260924 03:22:06.288961 17756 raft_consensus.cc:374] T ede60edd66d942dd88f89179c7bc6111 P 82c9a19bc3f44bf0bf0739243ee16066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:06.289058 17756 raft_consensus.cc:729] T ede60edd66d942dd88f89179c7bc6111 P 82c9a19bc3f44bf0bf0739243ee16066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c9a19bc3f44bf0bf0739243ee16066, State: Initialized, Role: FOLLOWER
I20260924 03:22:06.289701 17756 consensus_queue.cc:260] T ede60edd66d942dd88f89179c7bc6111 P 82c9a19bc3f44bf0bf0739243ee16066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c17e6da2e044dffbd40657771620fb5" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38003 } } peers { permanent_uuid: "82c9a19bc3f44bf0bf0739243ee16066" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46457 } } peers { permanent_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 39379 } }
I20260924 03:22:06.290012 17754 ts_tablet_manager.cc:1428] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260924 03:22:06.291604 17578 heartbeater.cc:503] Master 127.10.12.190:37301 was elected leader, sending a full tablet report...
I20260924 03:22:06.294603 17719 heartbeater.cc:503] Master 127.10.12.190:37301 was elected leader, sending a full tablet report...
I20260924 03:22:06.295483 17756 ts_tablet_manager.cc:1428] T ede60edd66d942dd88f89179c7bc6111 P 82c9a19bc3f44bf0bf0739243ee16066: Time spent starting tablet: real 0.018s user 0.013s sys 0.005s
I20260924 03:22:06.300170 17755 raft_consensus.cc:348] T ede60edd66d942dd88f89179c7bc6111 P 8c17e6da2e044dffbd40657771620fb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c17e6da2e044dffbd40657771620fb5" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38003 } } peers { permanent_uuid: "82c9a19bc3f44bf0bf0739243ee16066" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46457 } } peers { permanent_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 39379 } }
I20260924 03:22:06.300679 17755 raft_consensus.cc:374] T ede60edd66d942dd88f89179c7bc6111 P 8c17e6da2e044dffbd40657771620fb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:06.300796 17755 raft_consensus.cc:729] T ede60edd66d942dd88f89179c7bc6111 P 8c17e6da2e044dffbd40657771620fb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c17e6da2e044dffbd40657771620fb5, State: Initialized, Role: FOLLOWER
I20260924 03:22:06.301579 17755 consensus_queue.cc:260] T ede60edd66d942dd88f89179c7bc6111 P 8c17e6da2e044dffbd40657771620fb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c17e6da2e044dffbd40657771620fb5" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38003 } } peers { permanent_uuid: "82c9a19bc3f44bf0bf0739243ee16066" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46457 } } peers { permanent_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 39379 } }
I20260924 03:22:06.303575 17755 ts_tablet_manager.cc:1428] T ede60edd66d942dd88f89179c7bc6111 P 8c17e6da2e044dffbd40657771620fb5: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
W20260924 03:22:06.345449 17721 tablet.cc:2369] T ede60edd66d942dd88f89179c7bc6111 P 82c9a19bc3f44bf0bf0739243ee16066: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:22:06.396242 17440 tablet.cc:2369] T ede60edd66d942dd88f89179c7bc6111 P 8c17e6da2e044dffbd40657771620fb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:22:06.478464 17760 raft_consensus.cc:482] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:22:06.478786 17760 raft_consensus.cc:504] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c17e6da2e044dffbd40657771620fb5" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38003 } } peers { permanent_uuid: "82c9a19bc3f44bf0bf0739243ee16066" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46457 } } peers { permanent_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 39379 } }
I20260924 03:22:06.481240 17760 leader_election.cc:288] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c17e6da2e044dffbd40657771620fb5 (127.10.12.129:38003), 82c9a19bc3f44bf0bf0739243ee16066 (127.10.12.131:46457)
I20260924 03:22:06.490406 17749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:06.481904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:55073 (local address 127.10.12.129:38003)
0924 03:22:06.482133 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:06.482139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:06.483297 (+ 1158us) server_negotiation.cc:407] Connection header received
0924 03:22:06.483473 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:06.483478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:06.483574 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:06.483652 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:06.484412 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.485281 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.488060 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.489335 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.489514 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:06.489591 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:06.490001 (+ 410us) server_negotiation.cc:299] Negotiation successful
0924 03:22:06.490151 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 03:22:06.491182 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:06.482465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:38003 (local address 127.10.12.130:55073)
0924 03:22:06.483174 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:06.483194 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:06.483329 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:06.483656 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:06.483660 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:06.483677 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:06.484208 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:06.484218 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.485437 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.485441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:06.487834 (+ 2393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:06.487854 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.489610 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.489617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:06.489798 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:06.489836 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 03:22:06.490617 (+ 781us) client_negotiation.cc:240] Negotiation successful
0924 03:22:06.490782 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":47,"thread_start_us":440,"threads_started":1}
I20260924 03:22:06.492264 17392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede60edd66d942dd88f89179c7bc6111" candidate_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c17e6da2e044dffbd40657771620fb5" is_pre_election: true
I20260924 03:22:06.492653 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:06.482052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:57129 (local address 127.10.12.131:46457)
0924 03:22:06.482268 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:06.482274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:06.482928 (+ 654us) server_negotiation.cc:407] Connection header received
0924 03:22:06.483278 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:06.483284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:06.483374 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:06.483457 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:06.484344 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.485474 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.489684 (+ 4210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.490936 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.491097 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:06.491169 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:06.492149 (+ 980us) server_negotiation.cc:299] Negotiation successful
0924 03:22:06.492294 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 03:22:06.492841 17392 raft_consensus.cc:2454] T ede60edd66d942dd88f89179c7bc6111 P 8c17e6da2e044dffbd40657771620fb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cb91631ae5147c6aca234d1afe3b4cc in term 0.
I20260924 03:22:06.493623 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:06.481683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:46457 (local address 127.10.12.130:57129)
0924 03:22:06.482807 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:06.482828 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:06.483061 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:06.483623 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:06.483628 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:06.483654 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:06.484181 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:06.484199 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.485619 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.485624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:06.487834 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:06.487854 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.491736 (+ 3882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.491743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:06.491934 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:06.491994 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 03:22:06.492989 (+ 995us) client_negotiation.cc:240] Negotiation successful
0924 03:22:06.493196 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":104,"thread_start_us":635,"threads_started":1}
I20260924 03:22:06.493769 17467 leader_election.cc:302] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cb91631ae5147c6aca234d1afe3b4cc, 8c17e6da2e044dffbd40657771620fb5; no voters:
W20260924 03:22:06.494375 17580 tablet.cc:2369] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:22:06.494489 17760 raft_consensus.cc:2781] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:22:06.494711 17760 raft_consensus.cc:482] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:22:06.494840 17760 raft_consensus.cc:3037] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:06.497159 17673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede60edd66d942dd88f89179c7bc6111" candidate_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c9a19bc3f44bf0bf0739243ee16066" is_pre_election: true
I20260924 03:22:06.497784 17673 raft_consensus.cc:2454] T ede60edd66d942dd88f89179c7bc6111 P 82c9a19bc3f44bf0bf0739243ee16066 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cb91631ae5147c6aca234d1afe3b4cc in term 0.
I20260924 03:22:06.499174 17760 raft_consensus.cc:504] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c17e6da2e044dffbd40657771620fb5" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38003 } } peers { permanent_uuid: "82c9a19bc3f44bf0bf0739243ee16066" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46457 } } peers { permanent_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 39379 } }
I20260924 03:22:06.499825 17760 leader_election.cc:288] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc [CANDIDATE]: Term 1 election: Requested vote from peers 8c17e6da2e044dffbd40657771620fb5 (127.10.12.129:38003), 82c9a19bc3f44bf0bf0739243ee16066 (127.10.12.131:46457)
I20260924 03:22:06.500347 17673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede60edd66d942dd88f89179c7bc6111" candidate_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c9a19bc3f44bf0bf0739243ee16066"
I20260924 03:22:06.500373 17392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede60edd66d942dd88f89179c7bc6111" candidate_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c17e6da2e044dffbd40657771620fb5"
I20260924 03:22:06.500528 17673 raft_consensus.cc:3037] T ede60edd66d942dd88f89179c7bc6111 P 82c9a19bc3f44bf0bf0739243ee16066 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:06.500594 17392 raft_consensus.cc:3037] T ede60edd66d942dd88f89179c7bc6111 P 8c17e6da2e044dffbd40657771620fb5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:06.502754 17673 raft_consensus.cc:2454] T ede60edd66d942dd88f89179c7bc6111 P 82c9a19bc3f44bf0bf0739243ee16066 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cb91631ae5147c6aca234d1afe3b4cc in term 1.
I20260924 03:22:06.503206 17466 leader_election.cc:302] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cb91631ae5147c6aca234d1afe3b4cc, 82c9a19bc3f44bf0bf0739243ee16066; no voters:
I20260924 03:22:06.503515 17760 raft_consensus.cc:2781] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:22:06.503723 17392 raft_consensus.cc:2454] T ede60edd66d942dd88f89179c7bc6111 P 8c17e6da2e044dffbd40657771620fb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cb91631ae5147c6aca234d1afe3b4cc in term 1.
I20260924 03:22:06.504011 17760 raft_consensus.cc:686] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc [term 1 LEADER]: Becoming Leader. State: Replica: 3cb91631ae5147c6aca234d1afe3b4cc, State: Running, Role: LEADER
I20260924 03:22:06.504587 17760 consensus_queue.cc:237] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c17e6da2e044dffbd40657771620fb5" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38003 } } peers { permanent_uuid: "82c9a19bc3f44bf0bf0739243ee16066" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46457 } } peers { permanent_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 39379 } }
I20260924 03:22:06.509598 16452 catalog_manager.cc:5495] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cb91631ae5147c6aca234d1afe3b4cc (127.10.12.130). New cstate: current_term: 1 leader_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c17e6da2e044dffbd40657771620fb5" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82c9a19bc3f44bf0bf0739243ee16066" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 39379 } health_report { overall_health: HEALTHY } } }
I20260924 03:22:06.602197 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:06.596363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36216 (local address 127.10.12.130:39379)
0924 03:22:06.596566 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:06.596571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:06.596591 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:22:06.596683 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:06.596687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:06.596773 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:06.596906 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:22:06.597333 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.598092 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.599603 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.600108 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.600217 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:06.600873 (+ 656us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:22:06.601513 (+ 640us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:22:06.601630 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:06.601835 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 03:22:06.601923 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 03:22:06.611433 17673 raft_consensus.cc:1261] T ede60edd66d942dd88f89179c7bc6111 P 82c9a19bc3f44bf0bf0739243ee16066 [term 1 FOLLOWER]: Refusing update from remote peer 3cb91631ae5147c6aca234d1afe3b4cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:22:06.611435 17392 raft_consensus.cc:1261] T ede60edd66d942dd88f89179c7bc6111 P 8c17e6da2e044dffbd40657771620fb5 [term 1 FOLLOWER]: Refusing update from remote peer 3cb91631ae5147c6aca234d1afe3b4cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:22:06.612650 17760 consensus_queue.cc:1035] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c17e6da2e044dffbd40657771620fb5" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:22:06.613029 17765 consensus_queue.cc:1035] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "82c9a19bc3f44bf0bf0739243ee16066" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:22:06.650313 16452 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:22:06.650650 16452 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:22:06.660619 16452 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9fee3161a524093be6e2e6642cb28c3: Sending DeleteTablet for 3 replicas of tablet ede60edd66d942dd88f89179c7bc6111
I20260924 03:22:06.665715 17372 tablet_service.cc:1514] Processing DeleteTablet for tablet ede60edd66d942dd88f89179c7bc6111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:22:06 UTC) from {username='oryx', principal='oryx/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:36602
I20260924 03:22:06.666333 17512 tablet_service.cc:1514] Processing DeleteTablet for tablet ede60edd66d942dd88f89179c7bc6111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:22:06 UTC) from {username='oryx', principal='oryx/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:36212
I20260924 03:22:06.667104 17653 tablet_service.cc:1514] Processing DeleteTablet for tablet ede60edd66d942dd88f89179c7bc6111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:22:06 UTC) from {username='oryx', principal='oryx/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:54616
I20260924 03:22:06.667447 17778 tablet_replica.cc:330] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc: stopping tablet replica
I20260924 03:22:06.667884 17778 raft_consensus.cc:2229] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:22:06.668118 17779 tablet_replica.cc:330] T ede60edd66d942dd88f89179c7bc6111 P 82c9a19bc3f44bf0bf0739243ee16066: stopping tablet replica
I20260924 03:22:06.668583 17778 raft_consensus.cc:2258] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:22:06.669562 17779 raft_consensus.cc:2229] T ede60edd66d942dd88f89179c7bc6111 P 82c9a19bc3f44bf0bf0739243ee16066 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:22:06.669956 17779 raft_consensus.cc:2258] T ede60edd66d942dd88f89179c7bc6111 P 82c9a19bc3f44bf0bf0739243ee16066 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:22:06.673005 17778 ts_tablet_manager.cc:1905] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:22:06.676036 17777 tablet_replica.cc:330] T ede60edd66d942dd88f89179c7bc6111 P 8c17e6da2e044dffbd40657771620fb5: stopping tablet replica
I20260924 03:22:06.676504 17777 raft_consensus.cc:2229] T ede60edd66d942dd88f89179c7bc6111 P 8c17e6da2e044dffbd40657771620fb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:22:06.677066 17777 raft_consensus.cc:2258] T ede60edd66d942dd88f89179c7bc6111 P 8c17e6da2e044dffbd40657771620fb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:22:06.678222 17779 ts_tablet_manager.cc:1905] T ede60edd66d942dd88f89179c7bc6111 P 82c9a19bc3f44bf0bf0739243ee16066: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:22:06.679072 17777 ts_tablet_manager.cc:1905] T ede60edd66d942dd88f89179c7bc6111 P 8c17e6da2e044dffbd40657771620fb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:22:06.680153 17778 ts_tablet_manager.cc:1918] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:22:06.680333 17778 log.cc:1196] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/ede60edd66d942dd88f89179c7bc6111
I20260924 03:22:06.680955 17778 ts_tablet_manager.cc:1939] T ede60edd66d942dd88f89179c7bc6111 P 3cb91631ae5147c6aca234d1afe3b4cc: Deleting consensus metadata
I20260924 03:22:06.683071 16440 catalog_manager.cc:4841] TS 3cb91631ae5147c6aca234d1afe3b4cc (127.10.12.130:39379): tablet ede60edd66d942dd88f89179c7bc6111 (table test-table [id=ad2c24af36ff4e8181b199609fab677d]) successfully deleted
I20260924 03:22:06.686043 17777 ts_tablet_manager.cc:1918] T ede60edd66d942dd88f89179c7bc6111 P 8c17e6da2e044dffbd40657771620fb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:22:06.686332 17777 log.cc:1196] T ede60edd66d942dd88f89179c7bc6111 P 8c17e6da2e044dffbd40657771620fb5: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/wal/wals/ede60edd66d942dd88f89179c7bc6111
I20260924 03:22:06.686328 17779 ts_tablet_manager.cc:1918] T ede60edd66d942dd88f89179c7bc6111 P 82c9a19bc3f44bf0bf0739243ee16066: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:22:06.686517 17779 log.cc:1196] T ede60edd66d942dd88f89179c7bc6111 P 82c9a19bc3f44bf0bf0739243ee16066: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/ede60edd66d942dd88f89179c7bc6111
I20260924 03:22:06.687057 17779 ts_tablet_manager.cc:1939] T ede60edd66d942dd88f89179c7bc6111 P 82c9a19bc3f44bf0bf0739243ee16066: Deleting consensus metadata
I20260924 03:22:06.687072 17777 ts_tablet_manager.cc:1939] T ede60edd66d942dd88f89179c7bc6111 P 8c17e6da2e044dffbd40657771620fb5: Deleting consensus metadata
I20260924 03:22:06.689090 16438 catalog_manager.cc:4841] TS 82c9a19bc3f44bf0bf0739243ee16066 (127.10.12.131:46457): tablet ede60edd66d942dd88f89179c7bc6111 (table test-table [id=ad2c24af36ff4e8181b199609fab677d]) successfully deleted
I20260924 03:22:06.689229 16439 catalog_manager.cc:4841] TS 8c17e6da2e044dffbd40657771620fb5 (127.10.12.129:38003): tablet ede60edd66d942dd88f89179c7bc6111 (table test-table [id=ad2c24af36ff4e8181b199609fab677d]) successfully deleted
I20260924 03:22:06.690657 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 16421
I20260924 03:22:06.691979 16483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:06.703833 17735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:06.670368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50032 (local address 127.10.12.189:33891)
0924 03:22:06.670654 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:06.670661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:06.685516 (+ 14855us) server_negotiation.cc:407] Connection header received
0924 03:22:06.690411 (+ 4895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:06.690416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:06.690493 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:06.690575 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:06.692428 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.693632 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.695362 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.695760 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.695872 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:06.698668 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:06.698673 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:06.698676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:06.698714 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:06.701277 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:06.701985 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:06.701990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:06.701992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:06.702115 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:06.702473 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:06.702477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:06.702479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:06.702755 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:06.702996 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:06.703276 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 03:22:06.703511 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260924 03:22:06.717460 17781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:06.686117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42958 (local address 127.10.12.188:36159)
0924 03:22:06.689923 (+ 3806us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:06.689930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:06.703913 (+ 13983us) server_negotiation.cc:407] Connection header received
0924 03:22:06.704220 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:06.704225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:06.704314 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:06.704401 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:06.706113 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.706862 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.708633 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.709032 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.709145 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:06.711658 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:06.711663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:06.711667 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:06.711703 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:06.714503 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:06.715270 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:06.715276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:06.715280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:06.715456 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:06.715816 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:06.715821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:06.715824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:06.716224 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:06.716479 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:06.716801 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 03:22:06.717030 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3642,"thread_start_us":3541,"threads_started":1}
I20260924 03:22:06.887485 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 16421
I20260924 03:22:06.934487 17735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:06.919019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50046 (local address 127.10.12.189:33891)
0924 03:22:06.919204 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:06.919210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:06.919282 (+ 72us) server_negotiation.cc:407] Connection header received
0924 03:22:06.919521 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:06.919528 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:06.919625 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:06.919735 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:06.922036 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.922771 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.924611 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.925028 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.925142 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:06.928704 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:06.928710 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:06.928713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:06.928751 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:06.931590 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:06.932399 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:06.932404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:06.932406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:06.932595 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:06.933001 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:06.933006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:06.933010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:06.933293 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:06.933590 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:06.933941 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 03:22:06.934150 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 03:22:06.936929 17781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:06.920358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42960 (local address 127.10.12.188:36159)
0924 03:22:06.920598 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:06.920606 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:06.920627 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:22:06.920697 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:06.920702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:06.920793 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:06.920914 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:06.923170 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.924269 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.926273 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:06.926958 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:06.927102 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:06.930413 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:06.930420 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:06.930426 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:06.930471 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:06.934090 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:06.934870 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:06.934875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:06.934879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:06.935024 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:06.935434 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:06.935439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:06.935442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:06.935765 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:06.936029 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:06.936403 (+ 374us) server_negotiation.cc:299] Negotiation successful
0924 03:22:06.936626 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 03:22:07.638702 17789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:07.637893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.130:52475)
0924 03:22:07.638385 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:07.638515 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":254,"threads_started":1}
W20260924 03:22:07.639183 17578 heartbeater.cc:644] Failed to heartbeat to 127.10.12.190:37301 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
I20260924 03:22:07.668493 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:07.667621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.129:54701)
0924 03:22:07.668151 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:07.668302 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":233,"threads_started":1}
W20260924 03:22:07.668977 17438 heartbeater.cc:644] Failed to heartbeat to 127.10.12.190:37301 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
I20260924 03:22:07.669989 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:07.669193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.131:57269)
0924 03:22:07.669653 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:07.669791 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":215,"threads_started":1}
W20260924 03:22:07.670439 17719 heartbeater.cc:644] Failed to heartbeat to 127.10.12.190:37301 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
I20260924 03:22:08.158667 17792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e9fee3161a524093be6e2e6642cb28c3)
I20260924 03:22:08.158914 17792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: false } }
I20260924 03:22:08.159529 17793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e9fee3161a524093be6e2e6642cb28c3)
I20260924 03:22:08.159780 17792 leader_election.cc:288] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e9fee3161a524093be6e2e6642cb28c3 (127.10.12.190:37301), 85aaf56abb0d4474b9dd4bb4171f557c (127.10.12.189:33891)
I20260924 03:22:08.159804 17793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: false } }
I20260924 03:22:08.160933 17794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:08.160021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.0.0.1:35814)
0924 03:22:08.160486 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:08.160656 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":196,"threads_started":1}
I20260924 03:22:08.161507 17793 leader_election.cc:288] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e9fee3161a524093be6e2e6642cb28c3 (127.10.12.190:37301), 767765115d1d45b59d593a3a53cc529c (127.10.12.188:36159)
I20260924 03:22:08.162743 17797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:08.161768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.0.0.1:35826)
0924 03:22:08.162281 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:08.162487 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":266,"threads_started":1}
W20260924 03:22:08.162638 17238 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
W20260924 03:22:08.166952 17047 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
I20260924 03:22:08.171155 17794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:08.160619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:33891 (local address 127.0.0.1:50052)
0924 03:22:08.161266 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:08.161286 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:08.162533 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:08.163205 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:08.163211 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:08.163241 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:08.163509 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:08.163521 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:08.165321 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:08.165327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:08.167770 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:08.167790 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:08.170116 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:08.170123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:08.170347 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:08.170386 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 03:22:08.170512 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 03:22:08.170706 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":1,"thread_start_us":1678,"threads_started":1}
I20260924 03:22:08.172068 17796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:08.160733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50052 (local address 127.10.12.189:33891)
0924 03:22:08.162125 (+ 1392us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:08.162133 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:08.162160 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:22:08.162738 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:08.162744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:08.162831 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:08.162961 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:08.163729 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:08.165135 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:08.168027 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:08.169594 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:08.169729 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:08.170009 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:08.171439 (+ 1430us) server_negotiation.cc:299] Negotiation successful
0924 03:22:08.171635 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1233,"thread_start_us":1053,"threads_started":1}
I20260924 03:22:08.172617 17797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:08.162428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:36159 (local address 127.0.0.1:42962)
0924 03:22:08.162976 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:08.162993 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:08.163111 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:08.164012 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:08.164018 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:08.164052 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:08.164445 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:08.164456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:08.166828 (+ 2372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:08.166835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:08.169443 (+ 2608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:08.169464 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:08.171581 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:08.171589 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:08.171821 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:08.171864 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 03:22:08.171975 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 03:22:08.172231 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":31,"thread_start_us":4143,"threads_started":1}
I20260924 03:22:08.173182 17799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:08.162842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42962 (local address 127.10.12.188:36159)
0924 03:22:08.163651 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:08.163658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:08.163676 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:22:08.163738 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:08.163743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:08.163826 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:08.164591 (+ 765us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:08.164921 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:08.166316 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:08.169699 (+ 3383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:08.171219 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:08.172452 (+ 1233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:08.172591 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:08.172649 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 03:22:08.172814 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":530,"threads_started":1}
I20260924 03:22:08.173934 17261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "767765115d1d45b59d593a3a53cc529c" is_pre_election: true
I20260924 03:22:08.174140 17069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "767765115d1d45b59d593a3a53cc529c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" is_pre_election: true
I20260924 03:22:08.174332 17261 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85aaf56abb0d4474b9dd4bb4171f557c in term 1.
I20260924 03:22:08.174400 17069 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 767765115d1d45b59d593a3a53cc529c in term 1.
I20260924 03:22:08.174827 17797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:08.173933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.0.0.1:35832)
0924 03:22:08.174593 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:08.174669 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564}
I20260924 03:22:08.174842 17794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:08.174342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.0.0.1:35838)
0924 03:22:08.174576 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:08.174679 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 03:22:08.174853 17238 leader_election.cc:302] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 767765115d1d45b59d593a3a53cc529c, 85aaf56abb0d4474b9dd4bb4171f557c; no voters:
W20260924 03:22:08.175102 17047 leader_election.cc:334] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e9fee3161a524093be6e2e6642cb28c3 (127.10.12.190:37301): Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
I20260924 03:22:08.175184 17792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260924 03:22:08.175189 17238 leader_election.cc:334] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e9fee3161a524093be6e2e6642cb28c3 (127.10.12.190:37301): Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
I20260924 03:22:08.175303 17792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 1 FOLLOWER]: Starting leader election (detected failure of leader e9fee3161a524093be6e2e6642cb28c3)
I20260924 03:22:08.175411 17792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:22:08.175475 17047 leader_election.cc:302] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 767765115d1d45b59d593a3a53cc529c, 85aaf56abb0d4474b9dd4bb4171f557c; no voters: e9fee3161a524093be6e2e6642cb28c3
I20260924 03:22:08.175799 17793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 03:22:08.176010 17793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 1 FOLLOWER]: Starting leader election (detected failure of leader e9fee3161a524093be6e2e6642cb28c3)
I20260924 03:22:08.176116 17793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:22:08.177748 17792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: false } }
I20260924 03:22:08.178253 17793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: false } }
I20260924 03:22:08.178443 17792 leader_election.cc:288] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [CANDIDATE]: Term 2 election: Requested vote from peers e9fee3161a524093be6e2e6642cb28c3 (127.10.12.190:37301), 85aaf56abb0d4474b9dd4bb4171f557c (127.10.12.189:33891)
I20260924 03:22:08.178992 17793 leader_election.cc:288] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [CANDIDATE]: Term 2 election: Requested vote from peers e9fee3161a524093be6e2e6642cb28c3 (127.10.12.190:37301), 767765115d1d45b59d593a3a53cc529c (127.10.12.188:36159)
I20260924 03:22:08.179548 17794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:08.179025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.0.0.1:35840)
0924 03:22:08.179288 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:08.179382 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260924 03:22:08.179672 17797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:08.179224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.0.0.1:35848)
0924 03:22:08.179448 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:08.179536 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 03:22:08.179808 17261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "767765115d1d45b59d593a3a53cc529c"
I20260924 03:22:08.179840 17069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "767765115d1d45b59d593a3a53cc529c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "85aaf56abb0d4474b9dd4bb4171f557c"
I20260924 03:22:08.180138 17069 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 767765115d1d45b59d593a3a53cc529c in current term 2: Already voted for candidate 85aaf56abb0d4474b9dd4bb4171f557c in this term.
I20260924 03:22:08.180158 17261 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 85aaf56abb0d4474b9dd4bb4171f557c in current term 2: Already voted for candidate 767765115d1d45b59d593a3a53cc529c in this term.
I20260924 03:22:08.180807 17797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:08.180382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.0.0.1:35864)
0924 03:22:08.180551 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:08.180635 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 03:22:08.181133 17047 leader_election.cc:334] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e9fee3161a524093be6e2e6642cb28c3 (127.10.12.190:37301): Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
I20260924 03:22:08.181388 17047 leader_election.cc:302] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85aaf56abb0d4474b9dd4bb4171f557c; no voters: 767765115d1d45b59d593a3a53cc529c, e9fee3161a524093be6e2e6642cb28c3
I20260924 03:22:08.181521 17794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:08.180391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.0.0.1:35852)
0924 03:22:08.181173 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:08.181301 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578}
I20260924 03:22:08.181844 17793 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260924 03:22:08.181838 17238 leader_election.cc:334] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e9fee3161a524093be6e2e6642cb28c3 (127.10.12.190:37301): Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
I20260924 03:22:08.182014 17238 leader_election.cc:302] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 767765115d1d45b59d593a3a53cc529c; no voters: 85aaf56abb0d4474b9dd4bb4171f557c, e9fee3161a524093be6e2e6642cb28c3
I20260924 03:22:08.182367 17792 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 03:22:08.640905 17802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:08.640106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.130:47275)
0924 03:22:08.640621 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:08.640717 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":296,"threads_started":1}
I20260924 03:22:08.670598 17803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:08.669777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.129:49751)
0924 03:22:08.670277 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:08.670382 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":252,"threads_started":1}
I20260924 03:22:08.672021 17804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:08.671191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.131:37453)
0924 03:22:08.671667 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:08.671775 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":267,"threads_started":1}
I20260924 03:22:09.642997 17805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:09.642130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.130:41559)
0924 03:22:09.642692 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:09.642800 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":269,"threads_started":1}
I20260924 03:22:09.672873 17806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:09.671893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.129:43109)
0924 03:22:09.672486 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:09.672642 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":267,"threads_started":1}
I20260924 03:22:09.673843 17807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:09.672974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.131:37713)
0924 03:22:09.673518 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:09.673635 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":366,"threads_started":1}
I20260924 03:22:09.763909 17808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:22:09.764237 17808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: false } }
I20260924 03:22:09.765019 17808 leader_election.cc:288] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e9fee3161a524093be6e2e6642cb28c3 (127.10.12.190:37301), 767765115d1d45b59d593a3a53cc529c (127.10.12.188:36159)
I20260924 03:22:09.766394 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:09.765287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.0.0.1:57860)
0924 03:22:09.765861 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:09.766065 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":311,"threads_started":1}
I20260924 03:22:09.766543 17261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "767765115d1d45b59d593a3a53cc529c" is_pre_election: true
I20260924 03:22:09.766929 17261 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85aaf56abb0d4474b9dd4bb4171f557c in term 2.
I20260924 03:22:09.767541 17047 leader_election.cc:302] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 767765115d1d45b59d593a3a53cc529c, 85aaf56abb0d4474b9dd4bb4171f557c; no voters:
I20260924 03:22:09.767841 17808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 03:22:09.768033 17808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:22:09.768203 17808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 2 FOLLOWER]: Advancing to term 3
I20260924 03:22:09.768432 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:09.768027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.0.0.1:57878)
0924 03:22:09.768185 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:09.768269 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 03:22:09.768625 17047 leader_election.cc:334] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e9fee3161a524093be6e2e6642cb28c3 (127.10.12.190:37301): Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
I20260924 03:22:09.770740 17808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: false } }
I20260924 03:22:09.771394 17808 leader_election.cc:288] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [CANDIDATE]: Term 3 election: Requested vote from peers e9fee3161a524093be6e2e6642cb28c3 (127.10.12.190:37301), 767765115d1d45b59d593a3a53cc529c (127.10.12.188:36159)
I20260924 03:22:09.772063 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:09.771579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.0.0.1:57888)
0924 03:22:09.771758 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:09.771865 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 03:22:09.772286 17261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "767765115d1d45b59d593a3a53cc529c"
I20260924 03:22:09.772547 17261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 2 FOLLOWER]: Advancing to term 3
I20260924 03:22:09.773047 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:09.772713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.0.0.1:57890)
0924 03:22:09.772850 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:09.772927 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 03:22:09.773197 17047 leader_election.cc:334] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e9fee3161a524093be6e2e6642cb28c3 (127.10.12.190:37301): Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
I20260924 03:22:09.774938 17261 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85aaf56abb0d4474b9dd4bb4171f557c in term 3.
I20260924 03:22:09.775386 17047 leader_election.cc:302] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 767765115d1d45b59d593a3a53cc529c, 85aaf56abb0d4474b9dd4bb4171f557c; no voters: e9fee3161a524093be6e2e6642cb28c3
I20260924 03:22:09.775662 17808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 3 FOLLOWER]: Leader election won for term 3
I20260924 03:22:09.776196 17808 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [term 3 LEADER]: Becoming Leader. State: Replica: 85aaf56abb0d4474b9dd4bb4171f557c, State: Running, Role: LEADER
I20260924 03:22:09.776623 17808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: false } }
I20260924 03:22:09.778383 17811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85aaf56abb0d4474b9dd4bb4171f557c. Latest consensus state: current_term: 3 leader_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: false } } }
I20260924 03:22:09.778700 17811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:09.779389 17813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:22:09.783499 17813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:22:09.784442 17813 catalog_manager.cc:1259] Loaded cluster ID: 9da1f5522cb640e3bc32d53489f76eb7
I20260924 03:22:09.784514 17813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:22:09.785753 17813 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:22:09.786580 17813 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c: Loaded TSK: 0
I20260924 03:22:09.787597 17813 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:22:09.856244 17060 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:22:09.864493 17060 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:22:09.869503 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:09.869014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.0.0.1:57904)
0924 03:22:09.869199 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:09.869325 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 03:22:09.870008 17261 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [term 3 FOLLOWER]: Refusing update from remote peer 85aaf56abb0d4474b9dd4bb4171f557c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 03:22:09.870523 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:09.870115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.0.0.1:57912)
0924 03:22:09.870304 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:09.870375 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260924 03:22:09.870760 17811 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [LEADER]: Connected to new peer: Peer: permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260924 03:22:09.871089 17047 consensus_peers.cc:487] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c -> Peer e9fee3161a524093be6e2e6642cb28c3 (127.10.12.190:37301): Couldn't send request to peer e9fee3161a524093be6e2e6642cb28c3. Status: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 03:22:09.873680 17815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85aaf56abb0d4474b9dd4bb4171f557c. Latest consensus state: current_term: 3 leader_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: false } } }
I20260924 03:22:09.873941 17815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:22:09.875618 17811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: false } } }
I20260924 03:22:09.875998 17811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:09.876646 17815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e9fee3161a524093be6e2e6642cb28c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37301 } } peers { permanent_uuid: "85aaf56abb0d4474b9dd4bb4171f557c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 33891 } attrs { promote: false } } peers { permanent_uuid: "767765115d1d45b59d593a3a53cc529c" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 36159 } attrs { promote: false } } }
I20260924 03:22:09.876947 17815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 767765115d1d45b59d593a3a53cc529c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:22:09.894341 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:09.886152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:39379 (local address 127.0.0.1:52544)
0924 03:22:09.886415 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:09.886440 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:09.886582 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:09.888769 (+ 2187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:09.888776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:09.888813 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:09.889157 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:09.889170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:09.890355 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:09.890359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:09.891803 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:09.891817 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:09.893395 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:09.893401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:09.893575 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:09.893614 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 03:22:09.893702 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 03:22:09.893859 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260924 03:22:09.895375 17821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:09.886336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52544 (local address 127.10.12.130:39379)
0924 03:22:09.888422 (+ 2086us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:09.888427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:09.888446 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:22:09.888523 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:09.888528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:09.888597 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:09.888704 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:09.889308 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:09.890197 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:09.892011 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:09.893172 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:09.894636 (+ 1464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:09.894770 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:09.894835 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 03:22:09.895005 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1967,"thread_start_us":1814,"threads_started":1}
I20260924 03:22:09.897534 17512 tablet_service.cc:1467] Processing CreateTablet for tablet f6879ba2988141eb93272089b4a1a621 (DEFAULT_TABLE table=test-table [id=ad49ce43c45d45a5ba0a1044b1b57d14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:22:09.898145 17512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6879ba2988141eb93272089b4a1a621. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:09.901763 17823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:09.886824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40090 (local address 127.10.12.131:46457)
0924 03:22:09.887401 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:09.887406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:09.895016 (+ 7610us) server_negotiation.cc:407] Connection header received
0924 03:22:09.895310 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:09.895316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:09.895417 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:09.895515 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:09.896148 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:09.896932 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:09.899072 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:09.900402 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:09.900511 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:09.900631 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:09.901219 (+ 588us) server_negotiation.cc:299] Negotiation successful
0924 03:22:09.901396 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":254,"threads_started":1}
I20260924 03:22:09.902457 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:09.886487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:46457 (local address 127.0.0.1:40090)
0924 03:22:09.894909 (+ 8422us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:09.894930 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:09.895140 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:09.895774 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:09.895777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:09.895798 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:09.895995 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:09.896006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:09.897096 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:09.897100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:09.898876 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:09.898891 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:09.900798 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:09.900804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:09.901011 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:09.901051 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 03:22:09.901983 (+ 932us) client_negotiation.cc:240] Negotiation successful
0924 03:22:09.902146 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8315,"thread_start_us":15245,"threads_started":1}
I20260924 03:22:09.903509 17653 tablet_service.cc:1467] Processing CreateTablet for tablet f6879ba2988141eb93272089b4a1a621 (DEFAULT_TABLE table=test-table [id=ad49ce43c45d45a5ba0a1044b1b57d14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:22:09.904150 17653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6879ba2988141eb93272089b4a1a621. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:09.908738 17825 tablet_bootstrap.cc:492] T f6879ba2988141eb93272089b4a1a621 P 3cb91631ae5147c6aca234d1afe3b4cc: Bootstrap starting.
I20260924 03:22:09.911058 17825 tablet_bootstrap.cc:654] T f6879ba2988141eb93272089b4a1a621 P 3cb91631ae5147c6aca234d1afe3b4cc: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:09.913811 17826 tablet_bootstrap.cc:492] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066: Bootstrap starting.
I20260924 03:22:09.915742 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:09.895694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:38003 (local address 127.0.0.1:40998)
0924 03:22:09.901454 (+ 5760us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:09.901480 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:09.901635 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:09.908541 (+ 6906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:09.908549 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:09.908591 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:09.909261 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:09.909278 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:09.910893 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:09.910899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:09.913104 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:09.913125 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:09.914849 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:09.914856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:09.915071 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:09.915101 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 03:22:09.915201 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 03:22:09.915364 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5557}
I20260924 03:22:09.915748 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:09.896240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40998 (local address 127.10.12.129:38003)
0924 03:22:09.907809 (+ 11569us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:09.907817 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:09.907846 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:22:09.907943 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:09.907975 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:09.908098 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:09.908254 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:09.909543 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:09.910697 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:09.913302 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:09.914657 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:09.914792 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:09.914920 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:09.915237 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 03:22:09.915402 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11391,"thread_start_us":11779,"threads_started":1}
I20260924 03:22:09.916556 17825 tablet_bootstrap.cc:492] T f6879ba2988141eb93272089b4a1a621 P 3cb91631ae5147c6aca234d1afe3b4cc: No bootstrap required, opened a new log
I20260924 03:22:09.916734 17825 ts_tablet_manager.cc:1397] T f6879ba2988141eb93272089b4a1a621 P 3cb91631ae5147c6aca234d1afe3b4cc: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260924 03:22:09.916939 17372 tablet_service.cc:1467] Processing CreateTablet for tablet f6879ba2988141eb93272089b4a1a621 (DEFAULT_TABLE table=test-table [id=ad49ce43c45d45a5ba0a1044b1b57d14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:22:09.917506 17372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6879ba2988141eb93272089b4a1a621. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:09.917867 17825 raft_consensus.cc:348] T f6879ba2988141eb93272089b4a1a621 P 3cb91631ae5147c6aca234d1afe3b4cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 39379 } } peers { permanent_uuid: "82c9a19bc3f44bf0bf0739243ee16066" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46457 } } peers { permanent_uuid: "8c17e6da2e044dffbd40657771620fb5" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38003 } }
I20260924 03:22:09.918149 17825 raft_consensus.cc:374] T f6879ba2988141eb93272089b4a1a621 P 3cb91631ae5147c6aca234d1afe3b4cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:09.918267 17825 raft_consensus.cc:729] T f6879ba2988141eb93272089b4a1a621 P 3cb91631ae5147c6aca234d1afe3b4cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb91631ae5147c6aca234d1afe3b4cc, State: Initialized, Role: FOLLOWER
I20260924 03:22:09.918538 17825 consensus_queue.cc:260] T f6879ba2988141eb93272089b4a1a621 P 3cb91631ae5147c6aca234d1afe3b4cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 39379 } } peers { permanent_uuid: "82c9a19bc3f44bf0bf0739243ee16066" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46457 } } peers { permanent_uuid: "8c17e6da2e044dffbd40657771620fb5" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38003 } }
I20260924 03:22:09.919503 17825 ts_tablet_manager.cc:1428] T f6879ba2988141eb93272089b4a1a621 P 3cb91631ae5147c6aca234d1afe3b4cc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 03:22:09.920372 17805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:09.919841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.130:52627)
0924 03:22:09.920078 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:09.920197 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 03:22:09.922026 17805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:09.921094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.130:46953)
0924 03:22:09.921732 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:09.921834 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425}
I20260924 03:22:09.923475 17579 heartbeater.cc:503] Master 127.10.12.189:33891 was elected leader, sending a full tablet report...
I20260924 03:22:09.925897 17826 tablet_bootstrap.cc:654] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:09.925983 17829 tablet_bootstrap.cc:492] T f6879ba2988141eb93272089b4a1a621 P 8c17e6da2e044dffbd40657771620fb5: Bootstrap starting.
I20260924 03:22:09.928145 17829 tablet_bootstrap.cc:654] T f6879ba2988141eb93272089b4a1a621 P 8c17e6da2e044dffbd40657771620fb5: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:09.928747 17826 tablet_bootstrap.cc:492] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066: No bootstrap required, opened a new log
I20260924 03:22:09.928927 17826 ts_tablet_manager.cc:1397] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20260924 03:22:09.929797 17826 raft_consensus.cc:348] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 39379 } } peers { permanent_uuid: "82c9a19bc3f44bf0bf0739243ee16066" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46457 } } peers { permanent_uuid: "8c17e6da2e044dffbd40657771620fb5" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38003 } }
I20260924 03:22:09.930061 17826 raft_consensus.cc:374] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:09.930154 17826 raft_consensus.cc:729] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c9a19bc3f44bf0bf0739243ee16066, State: Initialized, Role: FOLLOWER
I20260924 03:22:09.930403 17826 consensus_queue.cc:260] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 39379 } } peers { permanent_uuid: "82c9a19bc3f44bf0bf0739243ee16066" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46457 } } peers { permanent_uuid: "8c17e6da2e044dffbd40657771620fb5" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38003 } }
I20260924 03:22:09.930596 17829 tablet_bootstrap.cc:492] T f6879ba2988141eb93272089b4a1a621 P 8c17e6da2e044dffbd40657771620fb5: No bootstrap required, opened a new log
I20260924 03:22:09.930747 17829 ts_tablet_manager.cc:1397] T f6879ba2988141eb93272089b4a1a621 P 8c17e6da2e044dffbd40657771620fb5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260924 03:22:09.931298 17826 ts_tablet_manager.cc:1428] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 03:22:09.932231 17807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:09.931720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.131:50663)
0924 03:22:09.931901 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:09.932033 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 03:22:09.933718 17720 heartbeater.cc:503] Master 127.10.12.189:33891 was elected leader, sending a full tablet report...
I20260924 03:22:09.934295 17807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:09.933837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.131:58631)
0924 03:22:09.933992 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:09.934091 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 03:22:09.934262 17829 raft_consensus.cc:348] T f6879ba2988141eb93272089b4a1a621 P 8c17e6da2e044dffbd40657771620fb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 39379 } } peers { permanent_uuid: "82c9a19bc3f44bf0bf0739243ee16066" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46457 } } peers { permanent_uuid: "8c17e6da2e044dffbd40657771620fb5" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38003 } }
I20260924 03:22:09.934556 17829 raft_consensus.cc:374] T f6879ba2988141eb93272089b4a1a621 P 8c17e6da2e044dffbd40657771620fb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:09.934674 17829 raft_consensus.cc:729] T f6879ba2988141eb93272089b4a1a621 P 8c17e6da2e044dffbd40657771620fb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c17e6da2e044dffbd40657771620fb5, State: Initialized, Role: FOLLOWER
I20260924 03:22:09.934984 17829 consensus_queue.cc:260] T f6879ba2988141eb93272089b4a1a621 P 8c17e6da2e044dffbd40657771620fb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 39379 } } peers { permanent_uuid: "82c9a19bc3f44bf0bf0739243ee16066" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46457 } } peers { permanent_uuid: "8c17e6da2e044dffbd40657771620fb5" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38003 } }
I20260924 03:22:09.936805 17829 ts_tablet_manager.cc:1428] T f6879ba2988141eb93272089b4a1a621 P 8c17e6da2e044dffbd40657771620fb5: Time spent starting tablet: real 0.006s user 0.000s sys 0.004s
I20260924 03:22:09.936856 17806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:09.936319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.129:43785)
0924 03:22:09.936506 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:09.936609 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 03:22:09.938134 17806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:09.937697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.129:39731)
0924 03:22:09.937860 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:09.937964 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 03:22:09.938736 17439 heartbeater.cc:503] Master 127.10.12.189:33891 was elected leader, sending a full tablet report...
I20260924 03:22:09.983003 17832 raft_consensus.cc:482] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:22:09.983357 17832 raft_consensus.cc:504] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 39379 } } peers { permanent_uuid: "82c9a19bc3f44bf0bf0739243ee16066" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46457 } } peers { permanent_uuid: "8c17e6da2e044dffbd40657771620fb5" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38003 } }
I20260924 03:22:09.985690 17832 leader_election.cc:288] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cb91631ae5147c6aca234d1afe3b4cc (127.10.12.130:39379), 8c17e6da2e044dffbd40657771620fb5 (127.10.12.129:38003)
I20260924 03:22:09.992551 17807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:09.986061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:39379 (local address 127.10.12.131:35221)
0924 03:22:09.986374 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:09.986394 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:09.986571 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:09.987342 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:09.987346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:09.987378 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:09.987693 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:09.987706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:09.988949 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:09.988955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:09.990447 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:09.990461 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:09.991698 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:09.991703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:09.991890 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:09.991920 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 03:22:09.992043 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 03:22:09.992227 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260924 03:22:09.992555 17821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:09.986593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:35221 (local address 127.10.12.130:39379)
0924 03:22:09.986871 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:09.986878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:09.986898 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:22:09.986985 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:09.986990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:09.987074 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:09.987169 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:09.987884 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:09.988808 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:09.990642 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:09.991516 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:09.991691 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:09.991781 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:09.992071 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 03:22:09.992224 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260924 03:22:09.992839 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:09.986928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:42873 (local address 127.10.12.129:38003)
0924 03:22:09.987153 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:09.987158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:09.987463 (+ 305us) server_negotiation.cc:407] Connection header received
0924 03:22:09.987724 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:09.987729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:09.987806 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:09.987883 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:09.988450 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:09.989223 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:09.990967 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:09.991786 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:09.991988 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:09.992068 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:09.992354 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 03:22:09.992542 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 03:22:09.992879 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:09.986156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:38003 (local address 127.10.12.131:42873)
0924 03:22:09.987352 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:09.987370 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:09.987580 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:09.987911 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:09.987915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:09.987938 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:09.988267 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:09.988278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:09.989361 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:09.989367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:09.990809 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:09.990822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:09.992033 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:09.992040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:09.992211 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:09.992235 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 03:22:09.992315 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 03:22:09.992456 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"thread_start_us":1173,"threads_started":1}
I20260924 03:22:09.993757 17392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6879ba2988141eb93272089b4a1a621" candidate_uuid: "82c9a19bc3f44bf0bf0739243ee16066" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c17e6da2e044dffbd40657771620fb5" is_pre_election: true
I20260924 03:22:09.994040 17392 raft_consensus.cc:2454] T f6879ba2988141eb93272089b4a1a621 P 8c17e6da2e044dffbd40657771620fb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82c9a19bc3f44bf0bf0739243ee16066 in term 0.
I20260924 03:22:09.993904 17532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6879ba2988141eb93272089b4a1a621" candidate_uuid: "82c9a19bc3f44bf0bf0739243ee16066" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" is_pre_election: true
I20260924 03:22:09.994419 17532 raft_consensus.cc:2454] T f6879ba2988141eb93272089b4a1a621 P 3cb91631ae5147c6aca234d1afe3b4cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82c9a19bc3f44bf0bf0739243ee16066 in term 0.
I20260924 03:22:09.994685 17608 leader_election.cc:302] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c9a19bc3f44bf0bf0739243ee16066, 8c17e6da2e044dffbd40657771620fb5; no voters:
I20260924 03:22:09.995144 17832 raft_consensus.cc:2781] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:22:09.995267 17832 raft_consensus.cc:482] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:22:09.995357 17832 raft_consensus.cc:3037] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:09.997831 17832 raft_consensus.cc:504] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 39379 } } peers { permanent_uuid: "82c9a19bc3f44bf0bf0739243ee16066" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46457 } } peers { permanent_uuid: "8c17e6da2e044dffbd40657771620fb5" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38003 } }
I20260924 03:22:09.998540 17832 leader_election.cc:288] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066 [CANDIDATE]: Term 1 election: Requested vote from peers 3cb91631ae5147c6aca234d1afe3b4cc (127.10.12.130:39379), 8c17e6da2e044dffbd40657771620fb5 (127.10.12.129:38003)
I20260924 03:22:09.999123 17532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6879ba2988141eb93272089b4a1a621" candidate_uuid: "82c9a19bc3f44bf0bf0739243ee16066" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb91631ae5147c6aca234d1afe3b4cc"
I20260924 03:22:09.999250 17392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6879ba2988141eb93272089b4a1a621" candidate_uuid: "82c9a19bc3f44bf0bf0739243ee16066" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c17e6da2e044dffbd40657771620fb5"
I20260924 03:22:09.999380 17532 raft_consensus.cc:3037] T f6879ba2988141eb93272089b4a1a621 P 3cb91631ae5147c6aca234d1afe3b4cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:09.999490 17392 raft_consensus.cc:3037] T f6879ba2988141eb93272089b4a1a621 P 8c17e6da2e044dffbd40657771620fb5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:10.002735 17392 raft_consensus.cc:2454] T f6879ba2988141eb93272089b4a1a621 P 8c17e6da2e044dffbd40657771620fb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82c9a19bc3f44bf0bf0739243ee16066 in term 1.
I20260924 03:22:10.002728 17532 raft_consensus.cc:2454] T f6879ba2988141eb93272089b4a1a621 P 3cb91631ae5147c6aca234d1afe3b4cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82c9a19bc3f44bf0bf0739243ee16066 in term 1.
I20260924 03:22:10.003356 17609 leader_election.cc:302] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cb91631ae5147c6aca234d1afe3b4cc, 82c9a19bc3f44bf0bf0739243ee16066; no voters:
I20260924 03:22:10.003695 17832 raft_consensus.cc:2781] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:22:10.004189 17832 raft_consensus.cc:686] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066 [term 1 LEADER]: Becoming Leader. State: Replica: 82c9a19bc3f44bf0bf0739243ee16066, State: Running, Role: LEADER
I20260924 03:22:10.004681 17832 consensus_queue.cc:237] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 39379 } } peers { permanent_uuid: "82c9a19bc3f44bf0bf0739243ee16066" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46457 } } peers { permanent_uuid: "8c17e6da2e044dffbd40657771620fb5" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38003 } }
I20260924 03:22:10.005563 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:10.005060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.131:54065)
0924 03:22:10.005269 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:10.005392 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 03:22:10.009043 17060 catalog_manager.cc:5495] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82c9a19bc3f44bf0bf0739243ee16066 (127.10.12.131). New cstate: current_term: 1 leader_uuid: "82c9a19bc3f44bf0bf0739243ee16066" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 39379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82c9a19bc3f44bf0bf0739243ee16066" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c17e6da2e044dffbd40657771620fb5" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38003 } health_report { overall_health: UNKNOWN } } }
I20260924 03:22:10.048408 17823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:10.042663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40100 (local address 127.10.12.131:46457)
0924 03:22:10.042918 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:10.042924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:10.042946 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:22:10.043081 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:10.043086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:10.043210 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:10.043371 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:22:10.043766 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:10.044568 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:10.045949 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:10.046413 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:10.046533 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:10.046981 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:22:10.047718 (+ 737us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:22:10.047859 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:10.047988 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 03:22:10.048087 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 03:22:10.053565 17392 raft_consensus.cc:1261] T f6879ba2988141eb93272089b4a1a621 P 8c17e6da2e044dffbd40657771620fb5 [term 1 FOLLOWER]: Refusing update from remote peer 82c9a19bc3f44bf0bf0739243ee16066: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:22:10.053956 17532 raft_consensus.cc:1261] T f6879ba2988141eb93272089b4a1a621 P 3cb91631ae5147c6aca234d1afe3b4cc [term 1 FOLLOWER]: Refusing update from remote peer 82c9a19bc3f44bf0bf0739243ee16066: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:22:10.054432 17835 consensus_queue.cc:1035] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c17e6da2e044dffbd40657771620fb5" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 38003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:22:10.054979 17832 consensus_queue.cc:1035] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cb91631ae5147c6aca234d1afe3b4cc" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:22:10.058979 17806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:10.058465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.129:37461)
0924 03:22:10.058659 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:10.058771 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 03:22:10.059098 17805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:10.058518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.130:58899)
0924 03:22:10.058794 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:10.058900 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260924 03:22:10.068403 17838 mvcc.cc:205] Tried to move back new op lower bound from 7332741652690644992 to 7332741652504133632. Current Snapshot: MvccSnapshot[applied={T|T < 7332741652690644992}]
I20260924 03:22:10.069361 17805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:10.068728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.130:37581)
0924 03:22:10.068938 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:10.069082 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 03:22:10.069851 17840 mvcc.cc:205] Tried to move back new op lower bound from 7332741652690644992 to 7332741652504133632. Current Snapshot: MvccSnapshot[applied={T|T < 7332741652690644992}]
I20260924 03:22:10.071811 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:10.071054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.131:57611)
0924 03:22:10.071420 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:10.071546 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223}
I20260924 03:22:10.074518 17839 mvcc.cc:205] Tried to move back new op lower bound from 7332741652690644992 to 7332741652504133632. Current Snapshot: MvccSnapshot[applied={T|T < 7332741652690644992}]
I20260924 03:22:10.077757 17806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:10.076769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.129:33483)
0924 03:22:10.077381 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:10.077499 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448}
I20260924 03:22:10.080878 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:10.077700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.10.12.131:38129)
0924 03:22:10.079455 (+ 1755us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:10.079632 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1597}
I20260924 03:22:10.088910 17059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:22:10.089308 17059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:22:10.096699 17059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85aaf56abb0d4474b9dd4bb4171f557c: Sending DeleteTablet for 3 replicas of tablet f6879ba2988141eb93272089b4a1a621
I20260924 03:22:10.097957 17512 tablet_service.cc:1514] Processing DeleteTablet for tablet f6879ba2988141eb93272089b4a1a621 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:22:10 UTC) from {username='oryx', principal='oryx/127.10.12.189@KRBTEST.COM'} at 127.0.0.1:52544
I20260924 03:22:10.098227 17372 tablet_service.cc:1514] Processing DeleteTablet for tablet f6879ba2988141eb93272089b4a1a621 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:22:10 UTC) from {username='oryx', principal='oryx/127.10.12.189@KRBTEST.COM'} at 127.0.0.1:40998
I20260924 03:22:10.098518 17653 tablet_service.cc:1514] Processing DeleteTablet for tablet f6879ba2988141eb93272089b4a1a621 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:22:10 UTC) from {username='oryx', principal='oryx/127.10.12.189@KRBTEST.COM'} at 127.0.0.1:40090
I20260924 03:22:10.101070 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 17223
I20260924 03:22:10.126356 17849 tablet_replica.cc:330] T f6879ba2988141eb93272089b4a1a621 P 8c17e6da2e044dffbd40657771620fb5: stopping tablet replica
I20260924 03:22:10.126731 17849 raft_consensus.cc:2229] T f6879ba2988141eb93272089b4a1a621 P 8c17e6da2e044dffbd40657771620fb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:22:10.127182 17849 raft_consensus.cc:2258] T f6879ba2988141eb93272089b4a1a621 P 8c17e6da2e044dffbd40657771620fb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:22:10.129510 17849 ts_tablet_manager.cc:1905] T f6879ba2988141eb93272089b4a1a621 P 8c17e6da2e044dffbd40657771620fb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:22:10.131709 17848 tablet_replica.cc:330] T f6879ba2988141eb93272089b4a1a621 P 3cb91631ae5147c6aca234d1afe3b4cc: stopping tablet replica
I20260924 03:22:10.132061 17850 tablet_replica.cc:330] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066: stopping tablet replica
I20260924 03:22:10.132078 17848 raft_consensus.cc:2229] T f6879ba2988141eb93272089b4a1a621 P 3cb91631ae5147c6aca234d1afe3b4cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:22:10.132340 17850 raft_consensus.cc:2229] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:22:10.132508 17848 raft_consensus.cc:2258] T f6879ba2988141eb93272089b4a1a621 P 3cb91631ae5147c6aca234d1afe3b4cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:22:10.133262 17850 raft_consensus.cc:2258] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:22:10.134253 17848 ts_tablet_manager.cc:1905] T f6879ba2988141eb93272089b4a1a621 P 3cb91631ae5147c6aca234d1afe3b4cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:22:10.137375 17850 ts_tablet_manager.cc:1905] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:22:10.140199 17849 ts_tablet_manager.cc:1918] T f6879ba2988141eb93272089b4a1a621 P 8c17e6da2e044dffbd40657771620fb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:22:10.140621 17849 log.cc:1196] T f6879ba2988141eb93272089b4a1a621 P 8c17e6da2e044dffbd40657771620fb5: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-0/wal/wals/f6879ba2988141eb93272089b4a1a621
I20260924 03:22:10.141461 17849 ts_tablet_manager.cc:1939] T f6879ba2988141eb93272089b4a1a621 P 8c17e6da2e044dffbd40657771620fb5: Deleting consensus metadata
I20260924 03:22:10.143268 17848 ts_tablet_manager.cc:1918] T f6879ba2988141eb93272089b4a1a621 P 3cb91631ae5147c6aca234d1afe3b4cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:22:10.143251 17046 catalog_manager.cc:4841] TS 8c17e6da2e044dffbd40657771620fb5 (127.10.12.129:38003): tablet f6879ba2988141eb93272089b4a1a621 (table test-table [id=ad49ce43c45d45a5ba0a1044b1b57d14]) successfully deleted
I20260924 03:22:10.143447 17848 log.cc:1196] T f6879ba2988141eb93272089b4a1a621 P 3cb91631ae5147c6aca234d1afe3b4cc: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/f6879ba2988141eb93272089b4a1a621
I20260924 03:22:10.143903 17848 ts_tablet_manager.cc:1939] T f6879ba2988141eb93272089b4a1a621 P 3cb91631ae5147c6aca234d1afe3b4cc: Deleting consensus metadata
I20260924 03:22:10.145447 17047 catalog_manager.cc:4841] TS 3cb91631ae5147c6aca234d1afe3b4cc (127.10.12.130:39379): tablet f6879ba2988141eb93272089b4a1a621 (table test-table [id=ad49ce43c45d45a5ba0a1044b1b57d14]) successfully deleted
I20260924 03:22:10.148733 17850 ts_tablet_manager.cc:1918] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:22:10.148931 17850 log.cc:1196] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/f6879ba2988141eb93272089b4a1a621
I20260924 03:22:10.149678 17850 ts_tablet_manager.cc:1939] T f6879ba2988141eb93272089b4a1a621 P 82c9a19bc3f44bf0bf0739243ee16066: Deleting consensus metadata
I20260924 03:22:10.151446 17045 catalog_manager.cc:4841] TS 82c9a19bc3f44bf0bf0739243ee16066 (127.10.12.131:46457): tablet f6879ba2988141eb93272089b4a1a621 (table test-table [id=ad49ce43c45d45a5ba0a1044b1b57d14]) successfully deleted
I20260924 03:22:10.152685 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:10.102215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41008 (local address 127.10.12.129:38003)
0924 03:22:10.102456 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:10.102465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:10.128173 (+ 25708us) server_negotiation.cc:407] Connection header received
0924 03:22:10.128605 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:10.128612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:10.128724 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:10.128827 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:10.131582 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:10.132810 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:10.135110 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:10.136068 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:10.136218 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:10.141790 (+ 5572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:10.141819 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:10.141848 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:10.141905 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:10.146599 (+ 4694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:10.148085 (+ 1486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:10.148107 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:10.148110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:10.148432 (+ 322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:10.149412 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:10.149418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:10.149422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:10.150293 (+ 871us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:10.150850 (+ 557us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:10.151720 (+ 870us) server_negotiation.cc:299] Negotiation successful
0924 03:22:10.152222 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 03:22:10.154480 17432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:10.280395 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 17223
I20260924 03:22:10.298153 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 17449
I20260924 03:22:10.318778 17821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:10.299252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52554 (local address 127.10.12.130:39379)
0924 03:22:10.299467 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:10.299472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:10.299492 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:22:10.299730 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:10.299734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:10.299815 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:10.299915 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:10.302033 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:10.302982 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:10.305202 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:10.305865 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:10.305987 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:10.310135 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:10.310165 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:10.310193 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:10.310248 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:10.314421 (+ 4173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:10.315680 (+ 1259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:10.315688 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:10.315691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:10.315863 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:10.316419 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:10.316426 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:10.316430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:10.317173 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:10.317646 (+ 473us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:10.317997 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 03:22:10.318321 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260924 03:22:10.319785 17572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:10.374329 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:10.373689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.0.0.1:57918)
0924 03:22:10.373949 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:10.374092 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 03:22:10.378165 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:10.377560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:37301 (local address 127.0.0.1:57934)
0924 03:22:10.377813 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:10.377941 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 03:22:10.463798 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 17449
I20260924 03:22:10.487622 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 17590
I20260924 03:22:10.509370 17823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:10.489777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40114 (local address 127.10.12.131:46457)
0924 03:22:10.490010 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:10.490016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:10.490038 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:22:10.490121 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:10.490127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:10.490238 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:10.490348 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:10.492495 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:10.494060 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:10.496311 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:10.497058 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:10.497214 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:10.501329 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:10.501511 (+ 182us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:10.501534 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:10.501602 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:10.505731 (+ 4129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:10.506872 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:10.506881 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:10.506884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:10.507096 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:10.507603 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:10.507606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:10.507609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:10.508097 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:10.508339 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:10.508763 (+ 424us) server_negotiation.cc:299] Negotiation successful
0924 03:22:10.508990 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 03:22:10.510140 17713 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:10.643908 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 17590
I20260924 03:22:10.660364 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 16915
I20260924 03:22:10.661625 17090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:10.786038 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 16915
W20260924 03:22:10.803110 17235 connection.cc:535] server connection from 127.0.0.1:42962 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 03:22:10.803442 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 17015
I20260924 03:22:10.804579 17281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:10.925423 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 17015
2026-09-24T03:22:10Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11654 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:22:11 dist-test-slave-00qs krb5kdc[17861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:22:11 dist-test-slave-00qs krb5kdc[17861](info): set up 2 sockets
Sep 24 03:22:11 dist-test-slave-00qs krb5kdc[17861](info): commencing operation
krb5kdc: starting...
W20260924 03:22:11.968950 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:22:12 dist-test-slave-00qs krb5kdc[17861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:22:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:22:12Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.190.keytab.
Entry for principal oryx/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.190.keytab.
I20260924 03:22:12.200191 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:43939
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.190.keytab
--principal=oryx/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:43939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:22:12.537056 17874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:12.537391 17874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:12.537529 17874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:12.545053 17874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:22:12.545192 17874 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 03:22:12.545254 17874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:12.545341 17874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:12.545408 17874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:22:12.545493 17874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:22:12.555773 17874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:43939
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.190.keytab
--principal=oryx/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:43939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:12.557257 17874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:12.558632 17874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:12.571099 17880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:12.571128 17882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:12.571342 17879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:12.571918 17874 server_base.cc:1034] running on GCE node
I20260924 03:22:12.572778 17874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:12.574337 17874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:12.575613 17874 hybrid_clock.cc:648] HybridClock initialized: now 1790220132575543 us; error 49 us; skew 500 ppm
I20260924 03:22:12.576071 17874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:12 dist-test-slave-00qs krb5kdc[17861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220132, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:12.579566 17874 init.cc:377] Logged in from keytab as oryx/127.10.12.190@KRBTEST.COM (short username oryx)
I20260924 03:22:12.581427 17874 webserver.cc:460] Webserver started at http://127.10.12.190:45383/ using document root <none> and password file <none>
I20260924 03:22:12.582192 17874 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:12.582291 17874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:12.582625 17874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:12.585808 17874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "809ce94c916f428f977b1e0c1a468934"
format_stamp: "Formatted at 2026-09-24 03:22:12 on dist-test-slave-00qs"
server_key: "e82e6141c43c2fe31cd4befe0a7c8ad0"
server_key_iv: "631e7438693335ddfcc411c06b4f6eca"
server_key_version: "encryptionkey@0"
I20260924 03:22:12.586659 17874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "809ce94c916f428f977b1e0c1a468934"
format_stamp: "Formatted at 2026-09-24 03:22:12 on dist-test-slave-00qs"
server_key: "e82e6141c43c2fe31cd4befe0a7c8ad0"
server_key_iv: "631e7438693335ddfcc411c06b4f6eca"
server_key_version: "encryptionkey@0"
I20260924 03:22:12.592559 17874 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 03:22:12.596726 17889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:12.598440 17874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:22:12.598645 17874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "809ce94c916f428f977b1e0c1a468934"
format_stamp: "Formatted at 2026-09-24 03:22:12 on dist-test-slave-00qs"
server_key: "e82e6141c43c2fe31cd4befe0a7c8ad0"
server_key_iv: "631e7438693335ddfcc411c06b4f6eca"
server_key_version: "encryptionkey@0"
I20260924 03:22:12.598899 17874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:12.613042 17874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:12.617492 17874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:12.617764 17874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:12.639693 17874 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:43939
I20260924 03:22:12.639745 17950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:43939 every 8 connection(s)
I20260924 03:22:12.641383 17874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:22:12.643172 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 17874
I20260924 03:22:12.643392 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:22:12.644304 10290 external_mini_cluster.cc:1428] Setting key c2044b6bee1605c936fe94d42056a0fa
I20260924 03:22:12.646708 17951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 03:22:12 dist-test-slave-00qs krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM
I20260924 03:22:12.660877 17951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934: Bootstrap starting.
I20260924 03:22:12.665077 17951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:12.666543 17951 log.cc:824] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934: Log is configured to *not* fsync() on all Append() calls
I20260924 03:22:12.670358 17951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934: No bootstrap required, opened a new log
I20260924 03:22:12.671543 17954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:12.646519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47714 (local address 127.10.12.190:43939)
0924 03:22:12.647383 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:12.647399 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:12.647450 (+ 51us) server_negotiation.cc:407] Connection header received
0924 03:22:12.649759 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:12.649792 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:12.650384 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:12.650907 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:12.652373 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:12.654446 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:12.655845 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:12.656946 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:12.657152 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:12.661873 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:12.661885 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:12.661907 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:12.661969 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:12.666078 (+ 4109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:12.667266 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:12.667274 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:12.667281 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:12.667483 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:12.668057 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:12.668063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:12.668066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:12.668833 (+ 767us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:12.669151 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:12.669603 (+ 452us) server_negotiation.cc:299] Negotiation successful
0924 03:22:12.670094 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":402,"threads_started":1}
I20260924 03:22:12.680047 17951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809ce94c916f428f977b1e0c1a468934" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 43939 } }
I20260924 03:22:12.681120 17951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:12.681286 17951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809ce94c916f428f977b1e0c1a468934, State: Initialized, Role: FOLLOWER
I20260924 03:22:12.682008 17951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809ce94c916f428f977b1e0c1a468934" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 43939 } }
I20260924 03:22:12.682283 17951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:22:12.682403 17951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:22:12.682576 17951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:12.684976 17951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809ce94c916f428f977b1e0c1a468934" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 43939 } }
I20260924 03:22:12.685616 17951 leader_election.cc:302] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 809ce94c916f428f977b1e0c1a468934; no voters:
I20260924 03:22:12.686352 17951 leader_election.cc:288] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:22:12.686590 17956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:22:12.687582 17956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934 [term 1 LEADER]: Becoming Leader. State: Replica: 809ce94c916f428f977b1e0c1a468934, State: Running, Role: LEADER
I20260924 03:22:12.688328 17956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809ce94c916f428f977b1e0c1a468934" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 43939 } }
I20260924 03:22:12.689028 17951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:22:12.691380 17957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "809ce94c916f428f977b1e0c1a468934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809ce94c916f428f977b1e0c1a468934" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 43939 } } }
I20260924 03:22:12.691794 17957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934 [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:12.692184 17958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 809ce94c916f428f977b1e0c1a468934. Latest consensus state: current_term: 1 leader_uuid: "809ce94c916f428f977b1e0c1a468934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809ce94c916f428f977b1e0c1a468934" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 43939 } } }
I20260924 03:22:12.692600 17958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934 [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:12.695634 17962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:22:12.701339 17962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:22:12.713289 17962 catalog_manager.cc:1347] Generated new cluster ID: 7aeaf3c7aaa945dfbeb9f06b54e89ae2
I20260924 03:22:12.713475 17962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:22:12.728852 17962 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:22:12.730832 17962 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:22:12.741122 17962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934: Generated new TSK 0
I20260924 03:22:12.742337 17962 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.129.keytab.
Entry for principal oryx/127.10.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.129.keytab.
I20260924 03:22:12.805135 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:43939
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.129.keytab
--principal=oryx/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:22:13.168179 17977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:13.168476 17977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:13.168619 17977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:13.176330 17977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:13.176461 17977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:13.176571 17977 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:22:13.176663 17977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:22:13.187196 17977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.129.keytab
--principal=oryx/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.12.190:43939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:13.188642 17977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:13.190083 17977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:13.203698 17985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:13.204032 17982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:13.204149 17983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:13.204728 17977 server_base.cc:1034] running on GCE node
I20260924 03:22:13.205384 17977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:13.206458 17977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:13.207702 17977 hybrid_clock.cc:648] HybridClock initialized: now 1790220133207619 us; error 58 us; skew 500 ppm
I20260924 03:22:13.208058 17977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:13 dist-test-slave-00qs krb5kdc[17861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220133, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:13.211434 17977 init.cc:377] Logged in from keytab as oryx/127.10.12.129@KRBTEST.COM (short username oryx)
I20260924 03:22:13.213167 17977 webserver.cc:460] Webserver started at http://127.10.12.129:42165/ using document root <none> and password file <none>
I20260924 03:22:13.213909 17977 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:13.214027 17977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:13.214350 17977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:13.217604 17977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "19fb8fb17c974ab189c484b2b39240dc"
format_stamp: "Formatted at 2026-09-24 03:22:13 on dist-test-slave-00qs"
server_key: "bc74062455665c94e42e25ab9cd4aea6"
server_key_iv: "eb621d8295a4747fc546926f251d8437"
server_key_version: "encryptionkey@0"
I20260924 03:22:13.218495 17977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "19fb8fb17c974ab189c484b2b39240dc"
format_stamp: "Formatted at 2026-09-24 03:22:13 on dist-test-slave-00qs"
server_key: "bc74062455665c94e42e25ab9cd4aea6"
server_key_iv: "eb621d8295a4747fc546926f251d8437"
server_key_version: "encryptionkey@0"
I20260924 03:22:13.224784 17977 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260924 03:22:13.229068 17992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:13.230792 17977 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 03:22:13.231006 17977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "19fb8fb17c974ab189c484b2b39240dc"
format_stamp: "Formatted at 2026-09-24 03:22:13 on dist-test-slave-00qs"
server_key: "bc74062455665c94e42e25ab9cd4aea6"
server_key_iv: "eb621d8295a4747fc546926f251d8437"
server_key_version: "encryptionkey@0"
I20260924 03:22:13.231258 17977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:13.256095 17977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:13.260870 17977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:13.261150 17977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:13.264120 17977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:22:13.264240 17977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:13.264387 17977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:22:13.264477 17977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:22:13 dist-test-slave-00qs krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220133, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM
I20260924 03:22:13.291512 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:13.264637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:54181 (local address 127.10.12.190:43939)
0924 03:22:13.265382 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:13.265386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:13.266672 (+ 1286us) server_negotiation.cc:407] Connection header received
0924 03:22:13.268896 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:13.268903 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:13.269003 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:13.269124 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:13.272008 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:13.272945 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:13.276199 (+ 3254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:13.276727 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:13.276893 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:13.283129 (+ 6236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:13.283136 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:13.283140 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:13.283196 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:13.287445 (+ 4249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:13.288464 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:13.288469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:13.288471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:13.288628 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:13.289192 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:13.289196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:13.289198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:13.289504 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:13.289649 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:13.290681 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0924 03:22:13.291021 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":555,"threads_started":1}
I20260924 03:22:13.292740 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:13.265442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:43939 (local address 127.10.12.129:54181)
0924 03:22:13.266501 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:13.266540 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:13.268567 (+ 2027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:13.269498 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:13.269511 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:13.270493 (+ 982us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:13.271690 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:13.271730 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:13.273182 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:13.273190 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:13.275926 (+ 2736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:13.275981 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:13.277268 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:13.277277 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:13.277605 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:13.278470 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:13.278515 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:13.282607 (+ 4092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:13.287668 (+ 5061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:13.287678 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:13.287696 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:13.288302 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:13.288757 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:13.288761 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:13.288768 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:13.289054 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:13.289644 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:13.289654 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:13.290119 (+ 465us) client_negotiation.cc:769] Sending connection context
0924 03:22:13.290507 (+ 388us) client_negotiation.cc:240] Negotiation successful
0924 03:22:13.291029 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":576,"threads_started":1}
I20260924 03:22:13.316390 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:13.299929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:46201 (local address 127.10.12.190:43939)
0924 03:22:13.300150 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:13.300155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:13.300744 (+ 589us) server_negotiation.cc:407] Connection header received
0924 03:22:13.300929 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:13.300935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:13.301010 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:13.301093 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:13.302144 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:13.302974 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:13.305056 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:13.305660 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:13.305798 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:13.309259 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:13.309266 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:13.309271 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:13.309323 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:13.312708 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:13.313861 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:13.313867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:13.313870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:13.314050 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:13.314547 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:13.314552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:13.314556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:13.314970 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:13.315141 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:13.315733 (+ 592us) server_negotiation.cc:299] Negotiation successful
0924 03:22:13.316029 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 03:22:13.316634 18066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:13.300039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:43939 (local address 127.10.12.129:46201)
0924 03:22:13.300626 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:13.300646 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:13.300804 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:13.301138 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:13.301143 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:13.301696 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:13.301995 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:13.302006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:13.303121 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:13.303127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:13.304831 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:13.304851 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:13.305894 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:13.305902 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:13.306433 (+ 531us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:13.307435 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:13.307468 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:13.308973 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:13.312915 (+ 3942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:13.312921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:13.312926 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:13.313686 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:13.314190 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:13.314199 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:13.314204 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:13.314400 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:13.315106 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:13.315115 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:13.315397 (+ 282us) client_negotiation.cc:769] Sending connection context
0924 03:22:13.315594 (+ 197us) client_negotiation.cc:240] Negotiation successful
0924 03:22:13.316080 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":314,"threads_started":1}
I20260924 03:22:13.320387 17977 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:34351
I20260924 03:22:13.320444 18112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:34351 every 8 connection(s)
I20260924 03:22:13.321633 17977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:22:13.322187 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 17977
I20260924 03:22:13.322417 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:22:13.323135 10290 external_mini_cluster.cc:1428] Setting key 965e2c0e7f4c76bece040f81b6fe848c
I20260924 03:22:13.323977 18113 heartbeater.cc:347] Connected to a master server at 127.10.12.190:43939
I20260924 03:22:13.324266 18113 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:13.325141 18113 heartbeater.cc:511] Master 127.10.12.190:43939 requested a full tablet report, sending...
I20260924 03:22:13.327296 17906 ts_manager.cc:194] Registered new tserver with Master: 19fb8fb17c974ab189c484b2b39240dc (127.10.12.129:34351)
I20260924 03:22:13.328928 17906 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.12.129@KRBTEST.COM'} at 127.10.12.129:54181
WARNING: no policy specified for oryx/127.10.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.130.keytab.
Entry for principal oryx/127.10.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.130.keytab.
I20260924 03:22:13.373579 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:43939
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.130.keytab
--principal=oryx/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:22:13.708134 18117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:13.708442 18117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:13.708565 18117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:13.716084 18117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:13.716210 18117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:13.716317 18117 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:22:13.716409 18117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:22:13.726878 18117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.130.keytab
--principal=oryx/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.12.190:43939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:13.728271 18117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:13.729677 18117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:13.743865 18123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:13.743894 18122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:13.743916 18125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:13.749814 18117 server_base.cc:1034] running on GCE node
I20260924 03:22:13.750548 18117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:13.751605 18117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:13.752852 18117 hybrid_clock.cc:648] HybridClock initialized: now 1790220133752761 us; error 68 us; skew 500 ppm
I20260924 03:22:13.753217 18117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:13 dist-test-slave-00qs krb5kdc[17861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220133, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:13.756472 18117 init.cc:377] Logged in from keytab as oryx/127.10.12.130@KRBTEST.COM (short username oryx)
I20260924 03:22:13.758062 18117 webserver.cc:460] Webserver started at http://127.10.12.130:42885/ using document root <none> and password file <none>
I20260924 03:22:13.758819 18117 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:13.758926 18117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:13.759253 18117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:13.762533 18117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "d02b12505f3343928b60d401fa75650e"
format_stamp: "Formatted at 2026-09-24 03:22:13 on dist-test-slave-00qs"
server_key: "739c240e716783a5821ef5a37a1f46af"
server_key_iv: "507d01f9cd181013f3f20a146107c693"
server_key_version: "encryptionkey@0"
I20260924 03:22:13.763399 18117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "d02b12505f3343928b60d401fa75650e"
format_stamp: "Formatted at 2026-09-24 03:22:13 on dist-test-slave-00qs"
server_key: "739c240e716783a5821ef5a37a1f46af"
server_key_iv: "507d01f9cd181013f3f20a146107c693"
server_key_version: "encryptionkey@0"
I20260924 03:22:13.769177 18117 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 03:22:13.772902 18132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:13.774405 18117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:22:13.774585 18117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "d02b12505f3343928b60d401fa75650e"
format_stamp: "Formatted at 2026-09-24 03:22:13 on dist-test-slave-00qs"
server_key: "739c240e716783a5821ef5a37a1f46af"
server_key_iv: "507d01f9cd181013f3f20a146107c693"
server_key_version: "encryptionkey@0"
I20260924 03:22:13.774813 18117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:13.787887 18117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:13.792564 18117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:13.792847 18117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:13.795642 18117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:22:13.795760 18117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:13.795892 18117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:22:13.795991 18117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:22:13 dist-test-slave-00qs krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220133, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM
I20260924 03:22:13.819644 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:13.796382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:44207 (local address 127.10.12.190:43939)
0924 03:22:13.796638 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:13.796645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:13.798193 (+ 1548us) server_negotiation.cc:407] Connection header received
0924 03:22:13.800609 (+ 2416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:13.800615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:13.800711 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:13.800804 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:13.803191 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:13.804591 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:13.807406 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:13.808125 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:13.808293 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:13.813269 (+ 4976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:13.813277 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:13.813282 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:13.813322 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:13.815845 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:13.816967 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:13.816973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:13.816977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:13.817137 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:13.817795 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:13.817800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:13.817804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:13.818108 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:13.818249 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:13.819045 (+ 796us) server_negotiation.cc:299] Negotiation successful
0924 03:22:13.819289 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 03:22:13.821065 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:13.797101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:43939 (local address 127.10.12.130:44207)
0924 03:22:13.798017 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:13.798069 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:13.800283 (+ 2214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:13.801070 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:13.801082 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:13.801863 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:13.802855 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:13.802884 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:13.804820 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:13.804826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:13.807172 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:13.807197 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:13.808381 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:13.808387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:13.808638 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:13.809555 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:13.809580 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:13.812978 (+ 3398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:13.816101 (+ 3123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:13.816112 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:13.816134 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:13.816829 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:13.817291 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:13.817299 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:13.817303 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:13.817670 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:13.818255 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:13.818265 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:13.818653 (+ 388us) client_negotiation.cc:769] Sending connection context
0924 03:22:13.818910 (+ 257us) client_negotiation.cc:240] Negotiation successful
0924 03:22:13.819491 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":422,"threads_started":1}
I20260924 03:22:13.843699 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:13.828393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:60395 (local address 127.10.12.190:43939)
0924 03:22:13.828627 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:13.828634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:13.829502 (+ 868us) server_negotiation.cc:407] Connection header received
0924 03:22:13.829643 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:13.829649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:13.829753 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:13.829872 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:13.831647 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:13.832650 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:13.834635 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:13.835170 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:13.835286 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:13.837734 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:13.837741 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:13.837747 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:13.837792 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:13.841036 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:13.841757 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:13.841763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:13.841767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:13.841908 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:13.842260 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:13.842265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:13.842269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:13.842617 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:13.842754 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:13.843151 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 03:22:13.843365 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 03:22:13.843849 18208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:13.828696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:43939 (local address 127.10.12.130:60395)
0924 03:22:13.829376 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:13.829400 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:13.829539 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:13.829925 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:13.829929 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:13.830798 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:13.831403 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:13.831422 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:13.832815 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:13.832818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:13.834474 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:13.834492 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:13.835364 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:13.835370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:13.835537 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:13.836369 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:13.836390 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:13.837514 (+ 1124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:13.841194 (+ 3680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:13.841200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:13.841207 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:13.841633 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:13.842016 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:13.842021 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:13.842026 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:13.842158 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:13.842735 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:13.842740 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:13.842923 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 03:22:13.843046 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 03:22:13.843395 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":393,"threads_started":1}
I20260924 03:22:13.849165 18117 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:33357
I20260924 03:22:13.849184 18251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:33357 every 8 connection(s)
I20260924 03:22:13.850342 18117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:22:13.852368 18252 heartbeater.cc:347] Connected to a master server at 127.10.12.190:43939
I20260924 03:22:13.852592 18252 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:13.853214 18252 heartbeater.cc:511] Master 127.10.12.190:43939 requested a full tablet report, sending...
I20260924 03:22:13.854554 17906 ts_manager.cc:194] Registered new tserver with Master: d02b12505f3343928b60d401fa75650e (127.10.12.130:33357)
I20260924 03:22:13.855520 17906 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.12.130@KRBTEST.COM'} at 127.10.12.130:44207
I20260924 03:22:13.856861 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 18117
I20260924 03:22:13.857000 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:22:13.857582 10290 external_mini_cluster.cc:1428] Setting key 59b60e245b4da98fa834df8950356c85
WARNING: no policy specified for oryx/127.10.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.131.keytab.
Entry for principal oryx/127.10.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.131.keytab.
I20260924 03:22:13.908290 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:43939
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.131.keytab
--principal=oryx/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:22:14.251222 18256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:14.251543 18256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:14.251670 18256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:14.259837 18256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:14.260012 18256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:14.260139 18256 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:22:14.260252 18256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:22:14.270987 18256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.131.keytab
--principal=oryx/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.12.190:43939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:14.272418 18256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:14.273864 18256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:14.288038 18261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:14.288111 18264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:14.288020 18262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:14.288816 18256 server_base.cc:1034] running on GCE node
I20260924 03:22:14.289517 18256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:14.290648 18256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:14.291893 18256 hybrid_clock.cc:648] HybridClock initialized: now 1790220134291821 us; error 65 us; skew 500 ppm
I20260924 03:22:14.292246 18256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:14 dist-test-slave-00qs krb5kdc[17861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220134, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:14.295598 18256 init.cc:377] Logged in from keytab as oryx/127.10.12.131@KRBTEST.COM (short username oryx)
I20260924 03:22:14.297472 18256 webserver.cc:460] Webserver started at http://127.10.12.131:38979/ using document root <none> and password file <none>
I20260924 03:22:14.298264 18256 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:14.298362 18256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:14.298674 18256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:14.301812 18256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "e95f7e6500af47b0b152fdb78860783a"
format_stamp: "Formatted at 2026-09-24 03:22:14 on dist-test-slave-00qs"
server_key: "691fe35fd2b1c07038aa58a74fe080c5"
server_key_iv: "e6899572e6ea0d1de17565bf707d3c02"
server_key_version: "encryptionkey@0"
I20260924 03:22:14.302683 18256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "e95f7e6500af47b0b152fdb78860783a"
format_stamp: "Formatted at 2026-09-24 03:22:14 on dist-test-slave-00qs"
server_key: "691fe35fd2b1c07038aa58a74fe080c5"
server_key_iv: "e6899572e6ea0d1de17565bf707d3c02"
server_key_version: "encryptionkey@0"
I20260924 03:22:14.308745 18256 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 03:22:14.312944 18271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:14.315114 18256 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:22:14.315311 18256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "e95f7e6500af47b0b152fdb78860783a"
format_stamp: "Formatted at 2026-09-24 03:22:14 on dist-test-slave-00qs"
server_key: "691fe35fd2b1c07038aa58a74fe080c5"
server_key_iv: "e6899572e6ea0d1de17565bf707d3c02"
server_key_version: "encryptionkey@0"
I20260924 03:22:14.315567 18256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:14.332444 18113 heartbeater.cc:503] Master 127.10.12.190:43939 was elected leader, sending a full tablet report...
I20260924 03:22:14.342511 18256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:14.347178 18256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:14.347481 18256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:14.350276 18256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:22:14.350395 18256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:14.350530 18256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:22:14.350595 18256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:22:14 dist-test-slave-00qs krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220134, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM
I20260924 03:22:14.375478 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:14.351064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:43265 (local address 127.10.12.190:43939)
0924 03:22:14.352350 (+ 1286us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:14.352355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:14.353087 (+ 732us) server_negotiation.cc:407] Connection header received
0924 03:22:14.355194 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:14.355200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:14.355290 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:14.355392 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:14.357642 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:14.358872 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:14.361427 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:14.361995 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:14.362126 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:14.367418 (+ 5292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:14.367426 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:14.367431 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:14.367509 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:14.371407 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:14.372406 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:14.372412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:14.372416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:14.372614 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:14.373159 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:14.373163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:14.373167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:14.373558 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:14.373732 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:14.374839 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0924 03:22:14.375088 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1172,"thread_start_us":1076,"threads_started":1}
I20260924 03:22:14.376748 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:14.352013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:43939 (local address 127.10.12.131:43265)
0924 03:22:14.352913 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:14.352961 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:14.354893 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:14.355664 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:14.355673 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:14.356489 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:14.357391 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:14.357430 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:14.359107 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:14.359112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:14.361232 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:14.361249 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:14.362318 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:14.362327 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:14.362727 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:14.363680 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:14.363723 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:14.367026 (+ 3303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:14.371636 (+ 4610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:14.371646 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:14.371663 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:14.372265 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:14.372768 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:14.372773 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:14.372775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:14.373032 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:14.373715 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:14.373725 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:14.374246 (+ 521us) client_negotiation.cc:769] Sending connection context
0924 03:22:14.374657 (+ 411us) client_negotiation.cc:240] Negotiation successful
0924 03:22:14.375248 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":392,"threads_started":1}
I20260924 03:22:14.397426 18256 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:33897
I20260924 03:22:14.397502 18391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:33897 every 8 connection(s)
I20260924 03:22:14.398976 18256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:22:14.401196 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 18256
I20260924 03:22:14.401376 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:22:14.401626 18392 heartbeater.cc:347] Connected to a master server at 127.10.12.190:43939
I20260924 03:22:14.401955 18392 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:14.401955 10290 external_mini_cluster.cc:1428] Setting key 4335c975f89bea5a1280728d65caaaef
I20260924 03:22:14.402575 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:14.384903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:56001 (local address 127.10.12.190:43939)
0924 03:22:14.385111 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:14.385118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:14.385849 (+ 731us) server_negotiation.cc:407] Connection header received
0924 03:22:14.386076 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:14.386081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:14.386254 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:14.386373 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:14.387805 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:14.388784 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:14.390744 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:14.391451 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:14.391603 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:14.394643 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:14.394650 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:14.394655 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:14.394703 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:14.398505 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:14.399595 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:14.399600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:14.399602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:14.400185 (+ 583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:14.400777 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:14.400782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:14.400785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:14.401203 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:14.401374 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:14.401912 (+ 538us) server_negotiation.cc:299] Negotiation successful
0924 03:22:14.402152 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 03:22:14.402637 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:14.384890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:43939 (local address 127.10.12.131:56001)
0924 03:22:14.385685 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:14.385710 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:14.385913 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:14.386435 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:14.386440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:14.387135 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:14.387573 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:14.387593 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:14.389001 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:14.389006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:14.390554 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:14.390576 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:14.391711 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:14.391719 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:14.391927 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:14.392832 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:14.392872 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:14.394254 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:14.398693 (+ 4439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:14.398699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:14.398703 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:14.399277 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:14.400363 (+ 1086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:14.400405 (+ 42us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:14.400409 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:14.400603 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:14.401424 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:14.401429 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:14.401660 (+ 231us) client_negotiation.cc:769] Sending connection context
0924 03:22:14.401792 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 03:22:14.402178 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":425,"threads_started":1}
I20260924 03:22:14.402961 18392 heartbeater.cc:511] Master 127.10.12.190:43939 requested a full tablet report, sending...
I20260924 03:22:14.404786 17905 ts_manager.cc:194] Registered new tserver with Master: e95f7e6500af47b0b152fdb78860783a (127.10.12.131:33897)
I20260924 03:22:14.405985 17905 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.12.131@KRBTEST.COM'} at 127.10.12.131:43265
I20260924 03:22:14.420292 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 03:22:14 dist-test-slave-00qs krb5kdc[17861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220134, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 03:22:14 dist-test-slave-00qs krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:14 dist-test-slave-00qs krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:14.858593 18252 heartbeater.cc:503] Master 127.10.12.190:43939 was elected leader, sending a full tablet report...
I20260924 03:22:15.408890 18392 heartbeater.cc:503] Master 127.10.12.190:43939 was elected leader, sending a full tablet report...
W20260924 03:22:15.445446 18400 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.979s user 0.005s sys 0.009s
W20260924 03:22:15.446755 18280 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:14.459559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.10.12.190:43939)
0924 03:22:14.459754 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:14.459759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:14.460215 (+ 456us) server_negotiation.cc:407] Connection header received
0924 03:22:14.460350 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:14.460355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:14.460427 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:14.460520 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:14.461874 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:14.462658 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:14.464268 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:14.464701 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:14.464820 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:15.446451 (+981631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47722: BlockingRecv error: recv got EOF from 127.0.0.1:47722 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 24 03:22:15 dist-test-slave-00qs krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220134, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM
I20260924 03:22:15.470372 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:15.451781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47738 (local address 127.10.12.190:43939)
0924 03:22:15.452096 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:15.452103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:15.452293 (+ 190us) server_negotiation.cc:407] Connection header received
0924 03:22:15.452477 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:15.452481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:15.452550 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:15.452634 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:15.453935 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.454679 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.456078 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.456537 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.456668 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:15.464728 (+ 8060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:15.464735 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:15.464738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:15.464782 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:15.467586 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:15.468473 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:15.468478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:15.468480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:15.468620 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:15.468989 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:15.468993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:15.468995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:15.469268 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:15.469502 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:15.469837 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 03:22:15.470082 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 03:22:15.475060 17905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:22:15.482378 17905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:22:15.509558 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:15.495626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49630 (local address 127.10.12.129:34351)
0924 03:22:15.500735 (+ 5109us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:15.500745 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:15.500774 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:22:15.500871 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:15.500959 (+ 88us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:15.501219 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:15.501427 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:15.502280 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.503476 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.506731 (+ 3255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.508284 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.508422 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:15.508645 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:15.508982 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 03:22:15.509175 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4958,"thread_start_us":4818,"threads_started":1}
I20260924 03:22:15.509625 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:15.497018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:33897 (local address 127.0.0.1:53930)
0924 03:22:15.500339 (+ 3321us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:15.500356 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:15.500474 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:15.501652 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:15.501656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:15.501686 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:15.502321 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:15.502335 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.504017 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.504023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:15.506517 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:15.506535 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.508783 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.508788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:15.508956 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:15.508999 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 03:22:15.509082 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 03:22:15.509216 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3209,"mutex_wait_us":102,"thread_start_us":3066,"threads_started":1}
I20260924 03:22:15.509648 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:15.495456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:34351 (local address 127.0.0.1:49630)
0924 03:22:15.497781 (+ 2325us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:15.497831 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:15.498019 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:15.501469 (+ 3450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:15.501473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:15.501510 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:15.502109 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:15.502127 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.503617 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.503625 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:15.506534 (+ 2909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:15.506543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.508497 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.508503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:15.508701 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:15.508764 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 03:22:15.508917 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 03:22:15.509142 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2058,"mutex_wait_us":208,"spinlock_wait_cycles":768,"thread_start_us":1935,"threads_started":1}
I20260924 03:22:15.509797 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:15.496118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57026 (local address 127.10.12.130:33357)
0924 03:22:15.497232 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:15.497242 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:15.500000 (+ 2758us) server_negotiation.cc:407] Connection header received
0924 03:22:15.500804 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:15.500813 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:15.501096 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:15.501234 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:15.503900 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.504968 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.507099 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.508305 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.508406 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:15.508647 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:15.509331 (+ 684us) server_negotiation.cc:299] Negotiation successful
0924 03:22:15.509513 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":949,"thread_start_us":828,"threads_started":1}
I20260924 03:22:15.510432 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:15.495638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:33357 (local address 127.0.0.1:57026)
0924 03:22:15.499841 (+ 4203us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:15.499861 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:15.500612 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:15.501462 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:15.501470 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:15.501518 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:15.503694 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:15.503709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.505112 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.505116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:15.506909 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:15.506922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.508887 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.508900 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:15.509107 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:15.509137 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 03:22:15.509998 (+ 861us) client_negotiation.cc:240] Negotiation successful
0924 03:22:15.510129 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4072,"mutex_wait_us":168,"spinlock_wait_cycles":8320,"thread_start_us":3944,"threads_started":1}
I20260924 03:22:15.510555 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:15.497587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53930 (local address 127.10.12.131:33897)
0924 03:22:15.501124 (+ 3537us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:15.501132 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:15.501158 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:22:15.501245 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:15.501252 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:15.501451 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:15.501650 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:15.502488 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.503859 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.506731 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.508284 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.508422 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:15.508645 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:15.510085 (+ 1440us) server_negotiation.cc:299] Negotiation successful
0924 03:22:15.510278 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3375,"thread_start_us":3181,"threads_started":1}
I20260924 03:22:15.513911 18322 tablet_service.cc:1467] Processing CreateTablet for tablet 5b7a41711a4d4ab893b5d7ace9d9d0f8 (DEFAULT_TABLE table=test-table [id=a8b9b703325c479299021c3cf7ae2504]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:22:15.514089 18043 tablet_service.cc:1467] Processing CreateTablet for tablet 5b7a41711a4d4ab893b5d7ace9d9d0f8 (DEFAULT_TABLE table=test-table [id=a8b9b703325c479299021c3cf7ae2504]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:22:15.514278 18182 tablet_service.cc:1467] Processing CreateTablet for tablet 5b7a41711a4d4ab893b5d7ace9d9d0f8 (DEFAULT_TABLE table=test-table [id=a8b9b703325c479299021c3cf7ae2504]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:22:15.515276 18322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b7a41711a4d4ab893b5d7ace9d9d0f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:15.515443 18043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b7a41711a4d4ab893b5d7ace9d9d0f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:15.515443 18182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b7a41711a4d4ab893b5d7ace9d9d0f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:15.527613 18417 tablet_bootstrap.cc:492] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P e95f7e6500af47b0b152fdb78860783a: Bootstrap starting.
I20260924 03:22:15.529876 18419 tablet_bootstrap.cc:492] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P d02b12505f3343928b60d401fa75650e: Bootstrap starting.
I20260924 03:22:15.530686 18418 tablet_bootstrap.cc:492] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc: Bootstrap starting.
I20260924 03:22:15.531412 18417 tablet_bootstrap.cc:654] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P e95f7e6500af47b0b152fdb78860783a: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:15.532856 18417 log.cc:824] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P e95f7e6500af47b0b152fdb78860783a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:22:15.534111 18418 tablet_bootstrap.cc:654] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:15.535075 18419 tablet_bootstrap.cc:654] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P d02b12505f3343928b60d401fa75650e: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:15.535416 18418 log.cc:824] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc: Log is configured to *not* fsync() on all Append() calls
I20260924 03:22:15.536371 18419 log.cc:824] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P d02b12505f3343928b60d401fa75650e: Log is configured to *not* fsync() on all Append() calls
I20260924 03:22:15.536944 18417 tablet_bootstrap.cc:492] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P e95f7e6500af47b0b152fdb78860783a: No bootstrap required, opened a new log
I20260924 03:22:15.537287 18417 ts_tablet_manager.cc:1397] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P e95f7e6500af47b0b152fdb78860783a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260924 03:22:15.539089 18418 tablet_bootstrap.cc:492] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc: No bootstrap required, opened a new log
I20260924 03:22:15.539319 18419 tablet_bootstrap.cc:492] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P d02b12505f3343928b60d401fa75650e: No bootstrap required, opened a new log
I20260924 03:22:15.539354 18418 ts_tablet_manager.cc:1397] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260924 03:22:15.539604 18419 ts_tablet_manager.cc:1397] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P d02b12505f3343928b60d401fa75650e: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260924 03:22:15.547811 18418 raft_consensus.cc:348] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.547821 18417 raft_consensus.cc:348] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.548285 18418 raft_consensus.cc:374] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:15.548352 18417 raft_consensus.cc:374] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:15.548399 18418 raft_consensus.cc:729] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fb8fb17c974ab189c484b2b39240dc, State: Initialized, Role: FOLLOWER
I20260924 03:22:15.548460 18417 raft_consensus.cc:729] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e95f7e6500af47b0b152fdb78860783a, State: Initialized, Role: FOLLOWER
I20260924 03:22:15.549064 18418 consensus_queue.cc:260] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.549124 18417 consensus_queue.cc:260] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P e95f7e6500af47b0b152fdb78860783a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.550137 18419 raft_consensus.cc:348] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.550593 18419 raft_consensus.cc:374] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:15.550693 18419 raft_consensus.cc:729] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02b12505f3343928b60d401fa75650e, State: Initialized, Role: FOLLOWER
I20260924 03:22:15.550760 18418 ts_tablet_manager.cc:1428] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260924 03:22:15.551296 18419 consensus_queue.cc:260] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P d02b12505f3343928b60d401fa75650e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.553247 18419 ts_tablet_manager.cc:1428] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P d02b12505f3343928b60d401fa75650e: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260924 03:22:15.558768 18417 ts_tablet_manager.cc:1428] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P e95f7e6500af47b0b152fdb78860783a: Time spent starting tablet: real 0.021s user 0.018s sys 0.000s
W20260924 03:22:15.576713 18114 tablet.cc:2369] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:22:15.577421 18423 raft_consensus.cc:482] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:22:15.577703 18423 raft_consensus.cc:504] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.579485 18423 leader_election.cc:288] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d02b12505f3343928b60d401fa75650e (127.10.12.130:33357), e95f7e6500af47b0b152fdb78860783a (127.10.12.131:33897)
I20260924 03:22:15.586017 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:15.580063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:32971 (local address 127.10.12.131:33897)
0924 03:22:15.580245 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:15.580250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:15.580987 (+ 737us) server_negotiation.cc:407] Connection header received
0924 03:22:15.581175 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:15.581179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:15.581252 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:15.581330 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:15.581823 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.582514 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.584358 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.585142 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.585294 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:15.585370 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:15.585648 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 03:22:15.585776 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 03:22:15.586151 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:15.580040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:33897 (local address 127.10.12.129:32971)
0924 03:22:15.580864 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:15.580885 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:15.581044 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:15.581373 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:15.581377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:15.581408 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:15.581681 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:15.581689 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.582649 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.582654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:15.584199 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:15.584213 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.585363 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.585369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:15.585498 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:15.585521 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 03:22:15.585606 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 03:22:15.585738 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":24,"thread_start_us":368,"threads_started":1}
I20260924 03:22:15.586503 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:15.580042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:50047 (local address 127.10.12.130:33357)
0924 03:22:15.580229 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:15.580235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:15.581028 (+ 793us) server_negotiation.cc:407] Connection header received
0924 03:22:15.581254 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:15.581258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:15.581558 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:15.581730 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:15.582386 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.583066 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.584811 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.585657 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.585801 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:15.585882 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:15.586142 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 03:22:15.586275 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 03:22:15.586627 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:15.580059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:33357 (local address 127.10.12.129:50047)
0924 03:22:15.580912 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:15.580926 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:15.581111 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:15.581859 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:15.581864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:15.581888 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:15.582208 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:15.582217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.583207 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.583211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:15.584656 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:15.584669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.585876 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.585883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:15.586006 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:15.586031 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 03:22:15.586112 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 03:22:15.586238 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":112,"spinlock_wait_cycles":25472,"thread_start_us":449,"threads_started":1}
I20260924 03:22:15.587157 18342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b7a41711a4d4ab893b5d7ace9d9d0f8" candidate_uuid: "19fb8fb17c974ab189c484b2b39240dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e95f7e6500af47b0b152fdb78860783a" is_pre_election: true
I20260924 03:22:15.587658 18342 raft_consensus.cc:2454] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19fb8fb17c974ab189c484b2b39240dc in term 0.
I20260924 03:22:15.587714 18206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b7a41711a4d4ab893b5d7ace9d9d0f8" candidate_uuid: "19fb8fb17c974ab189c484b2b39240dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d02b12505f3343928b60d401fa75650e" is_pre_election: true
I20260924 03:22:15.588255 18206 raft_consensus.cc:2454] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19fb8fb17c974ab189c484b2b39240dc in term 0.
I20260924 03:22:15.588289 17995 leader_election.cc:302] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19fb8fb17c974ab189c484b2b39240dc, e95f7e6500af47b0b152fdb78860783a; no voters:
I20260924 03:22:15.588658 18423 raft_consensus.cc:2781] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:22:15.588825 18423 raft_consensus.cc:482] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:22:15.588943 18423 raft_consensus.cc:3037] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:15.591276 18423 raft_consensus.cc:504] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.591902 18423 leader_election.cc:288] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc [CANDIDATE]: Term 1 election: Requested vote from peers d02b12505f3343928b60d401fa75650e (127.10.12.130:33357), e95f7e6500af47b0b152fdb78860783a (127.10.12.131:33897)
I20260924 03:22:15.592429 18206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b7a41711a4d4ab893b5d7ace9d9d0f8" candidate_uuid: "19fb8fb17c974ab189c484b2b39240dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d02b12505f3343928b60d401fa75650e"
I20260924 03:22:15.592648 18206 raft_consensus.cc:3037] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:15.592914 18342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b7a41711a4d4ab893b5d7ace9d9d0f8" candidate_uuid: "19fb8fb17c974ab189c484b2b39240dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e95f7e6500af47b0b152fdb78860783a"
I20260924 03:22:15.593159 18342 raft_consensus.cc:3037] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:15.595887 18206 raft_consensus.cc:2454] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P d02b12505f3343928b60d401fa75650e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19fb8fb17c974ab189c484b2b39240dc in term 1.
I20260924 03:22:15.596227 18342 raft_consensus.cc:2454] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P e95f7e6500af47b0b152fdb78860783a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19fb8fb17c974ab189c484b2b39240dc in term 1.
I20260924 03:22:15.596452 17994 leader_election.cc:302] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19fb8fb17c974ab189c484b2b39240dc, d02b12505f3343928b60d401fa75650e; no voters:
I20260924 03:22:15.596822 18423 raft_consensus.cc:2781] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:22:15.597270 18423 raft_consensus.cc:686] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc [term 1 LEADER]: Becoming Leader. State: Replica: 19fb8fb17c974ab189c484b2b39240dc, State: Running, Role: LEADER
I20260924 03:22:15.597764 18423 consensus_queue.cc:237] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.602742 17905 catalog_manager.cc:5495] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 19fb8fb17c974ab189c484b2b39240dc (127.10.12.129). New cstate: current_term: 1 leader_uuid: "19fb8fb17c974ab189c484b2b39240dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } health_report { overall_health: UNKNOWN } } }
W20260924 03:22:15.603261 18253 tablet.cc:2369] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P d02b12505f3343928b60d401fa75650e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:22:15.634073 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:15.628730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49638 (local address 127.10.12.129:34351)
0924 03:22:15.628903 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:15.628908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:15.629020 (+ 112us) server_negotiation.cc:407] Connection header received
0924 03:22:15.629215 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:15.629218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:15.629316 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:15.629451 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:22:15.629758 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.630492 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.631912 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.632412 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.632514 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:15.632940 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:22:15.633560 (+ 620us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:22:15.633672 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:15.633765 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 03:22:15.633840 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 03:22:15.642706 18206 raft_consensus.cc:1261] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P d02b12505f3343928b60d401fa75650e [term 1 FOLLOWER]: Refusing update from remote peer 19fb8fb17c974ab189c484b2b39240dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:22:15.642901 18342 raft_consensus.cc:1261] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P e95f7e6500af47b0b152fdb78860783a [term 1 FOLLOWER]: Refusing update from remote peer 19fb8fb17c974ab189c484b2b39240dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:22:15.643819 18428 consensus_queue.cc:1035] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:22:15.644299 18423 consensus_queue.cc:1035] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 03:22:15.652341 18393 tablet.cc:2369] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P e95f7e6500af47b0b152fdb78860783a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:22:15.675444 17905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:22:15.675729 17905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:22:15.679730 17905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934: Sending DeleteTablet for 3 replicas of tablet 5b7a41711a4d4ab893b5d7ace9d9d0f8
I20260924 03:22:15.680881 18043 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b7a41711a4d4ab893b5d7ace9d9d0f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:22:15 UTC) from {username='oryx', principal='oryx/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:49630
I20260924 03:22:15.681257 18182 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b7a41711a4d4ab893b5d7ace9d9d0f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:22:15 UTC) from {username='oryx', principal='oryx/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:57026
I20260924 03:22:15.681854 18440 tablet_replica.cc:330] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc: stopping tablet replica
I20260924 03:22:15.682341 18440 raft_consensus.cc:2229] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:22:15.682381 18322 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b7a41711a4d4ab893b5d7ace9d9d0f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:22:15 UTC) from {username='oryx', principal='oryx/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:53930
I20260924 03:22:15.683096 18440 raft_consensus.cc:2258] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:22:15.685829 18440 ts_tablet_manager.cc:1905] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:22:15.687832 17906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 03:22:15.688531 18441 tablet_replica.cc:330] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P d02b12505f3343928b60d401fa75650e: stopping tablet replica
W20260924 03:22:15.688618 17906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:22:15.688776 18442 tablet_replica.cc:330] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P e95f7e6500af47b0b152fdb78860783a: stopping tablet replica
I20260924 03:22:15.689013 18441 raft_consensus.cc:2229] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P d02b12505f3343928b60d401fa75650e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:22:15.689142 18442 raft_consensus.cc:2229] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P e95f7e6500af47b0b152fdb78860783a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:22:15.689565 18442 raft_consensus.cc:2258] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P e95f7e6500af47b0b152fdb78860783a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:22:15.689565 18441 raft_consensus.cc:2258] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P d02b12505f3343928b60d401fa75650e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:22:15.691799 18442 ts_tablet_manager.cc:1905] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P e95f7e6500af47b0b152fdb78860783a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:22:15.693768 18440 ts_tablet_manager.cc:1918] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:22:15.694100 18440 log.cc:1196] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-0/wal/wals/5b7a41711a4d4ab893b5d7ace9d9d0f8
I20260924 03:22:15.694758 18440 ts_tablet_manager.cc:1939] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P 19fb8fb17c974ab189c484b2b39240dc: Deleting consensus metadata
I20260924 03:22:15.697047 17890 catalog_manager.cc:4841] TS 19fb8fb17c974ab189c484b2b39240dc (127.10.12.129:34351): tablet 5b7a41711a4d4ab893b5d7ace9d9d0f8 (table test-table [id=a8b9b703325c479299021c3cf7ae2504]) successfully deleted
I20260924 03:22:15.698586 18182 tablet_service.cc:1467] Processing CreateTablet for tablet bb4b63bee6e24286b6bf7968dc01e0c4 (DEFAULT_TABLE table=test-table [id=af2f5c5cc0a74460a07b19fdce14cbfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:22:15.698593 18043 tablet_service.cc:1467] Processing CreateTablet for tablet bb4b63bee6e24286b6bf7968dc01e0c4 (DEFAULT_TABLE table=test-table [id=af2f5c5cc0a74460a07b19fdce14cbfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:22:15.699057 18182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb4b63bee6e24286b6bf7968dc01e0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:15.699056 18043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb4b63bee6e24286b6bf7968dc01e0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:15.699595 18442 ts_tablet_manager.cc:1918] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P e95f7e6500af47b0b152fdb78860783a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:22:15.699770 18442 log.cc:1196] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P e95f7e6500af47b0b152fdb78860783a: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/5b7a41711a4d4ab893b5d7ace9d9d0f8
I20260924 03:22:15.702306 18441 ts_tablet_manager.cc:1905] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P d02b12505f3343928b60d401fa75650e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:22:15.706197 18419 tablet_bootstrap.cc:492] T bb4b63bee6e24286b6bf7968dc01e0c4 P d02b12505f3343928b60d401fa75650e: Bootstrap starting.
I20260924 03:22:15.707926 18419 tablet_bootstrap.cc:654] T bb4b63bee6e24286b6bf7968dc01e0c4 P d02b12505f3343928b60d401fa75650e: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:15.709772 18441 ts_tablet_manager.cc:1918] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P d02b12505f3343928b60d401fa75650e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:22:15.710049 18442 ts_tablet_manager.cc:1939] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P e95f7e6500af47b0b152fdb78860783a: Deleting consensus metadata
I20260924 03:22:15.710146 18321 tablet_service.cc:1467] Processing CreateTablet for tablet bb4b63bee6e24286b6bf7968dc01e0c4 (DEFAULT_TABLE table=test-table [id=af2f5c5cc0a74460a07b19fdce14cbfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:22:15.710693 18321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb4b63bee6e24286b6bf7968dc01e0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:15.712209 18441 log.cc:1196] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P d02b12505f3343928b60d401fa75650e: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/5b7a41711a4d4ab893b5d7ace9d9d0f8
I20260924 03:22:15.712867 18441 ts_tablet_manager.cc:1939] T 5b7a41711a4d4ab893b5d7ace9d9d0f8 P d02b12505f3343928b60d401fa75650e: Deleting consensus metadata
I20260924 03:22:15.715019 17891 catalog_manager.cc:4841] TS d02b12505f3343928b60d401fa75650e (127.10.12.130:33357): tablet 5b7a41711a4d4ab893b5d7ace9d9d0f8 (table test-table [id=a8b9b703325c479299021c3cf7ae2504]) successfully deleted
I20260924 03:22:15.715719 17892 catalog_manager.cc:4841] TS e95f7e6500af47b0b152fdb78860783a (127.10.12.131:33897): tablet 5b7a41711a4d4ab893b5d7ace9d9d0f8 (table test-table [id=a8b9b703325c479299021c3cf7ae2504]) successfully deleted
I20260924 03:22:15.717844 18417 tablet_bootstrap.cc:492] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a: Bootstrap starting.
I20260924 03:22:15.719739 18417 tablet_bootstrap.cc:654] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:15.720108 18418 tablet_bootstrap.cc:492] T bb4b63bee6e24286b6bf7968dc01e0c4 P 19fb8fb17c974ab189c484b2b39240dc: Bootstrap starting.
I20260924 03:22:15.721823 18418 tablet_bootstrap.cc:654] T bb4b63bee6e24286b6bf7968dc01e0c4 P 19fb8fb17c974ab189c484b2b39240dc: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:15.722540 18419 tablet_bootstrap.cc:492] T bb4b63bee6e24286b6bf7968dc01e0c4 P d02b12505f3343928b60d401fa75650e: No bootstrap required, opened a new log
I20260924 03:22:15.722708 18419 ts_tablet_manager.cc:1397] T bb4b63bee6e24286b6bf7968dc01e0c4 P d02b12505f3343928b60d401fa75650e: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20260924 03:22:15.723644 18419 raft_consensus.cc:348] T bb4b63bee6e24286b6bf7968dc01e0c4 P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.724078 18419 raft_consensus.cc:374] T bb4b63bee6e24286b6bf7968dc01e0c4 P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:15.724224 18419 raft_consensus.cc:729] T bb4b63bee6e24286b6bf7968dc01e0c4 P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02b12505f3343928b60d401fa75650e, State: Initialized, Role: FOLLOWER
I20260924 03:22:15.724596 18419 consensus_queue.cc:260] T bb4b63bee6e24286b6bf7968dc01e0c4 P d02b12505f3343928b60d401fa75650e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.726388 18419 ts_tablet_manager.cc:1428] T bb4b63bee6e24286b6bf7968dc01e0c4 P d02b12505f3343928b60d401fa75650e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 03:22:15.728639 18417 tablet_bootstrap.cc:492] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a: No bootstrap required, opened a new log
I20260924 03:22:15.728710 18418 tablet_bootstrap.cc:492] T bb4b63bee6e24286b6bf7968dc01e0c4 P 19fb8fb17c974ab189c484b2b39240dc: No bootstrap required, opened a new log
I20260924 03:22:15.728929 18418 ts_tablet_manager.cc:1397] T bb4b63bee6e24286b6bf7968dc01e0c4 P 19fb8fb17c974ab189c484b2b39240dc: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20260924 03:22:15.728922 18417 ts_tablet_manager.cc:1397] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20260924 03:22:15.729808 18417 raft_consensus.cc:348] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.730120 18417 raft_consensus.cc:374] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:15.730268 18417 raft_consensus.cc:729] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e95f7e6500af47b0b152fdb78860783a, State: Initialized, Role: FOLLOWER
I20260924 03:22:15.730702 18417 consensus_queue.cc:260] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.730847 18418 raft_consensus.cc:348] T bb4b63bee6e24286b6bf7968dc01e0c4 P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.731112 18418 raft_consensus.cc:374] T bb4b63bee6e24286b6bf7968dc01e0c4 P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:15.731206 18418 raft_consensus.cc:729] T bb4b63bee6e24286b6bf7968dc01e0c4 P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fb8fb17c974ab189c484b2b39240dc, State: Initialized, Role: FOLLOWER
I20260924 03:22:15.731436 18417 ts_tablet_manager.cc:1428] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:22:15.731505 18418 consensus_queue.cc:260] T bb4b63bee6e24286b6bf7968dc01e0c4 P 19fb8fb17c974ab189c484b2b39240dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.732281 18418 ts_tablet_manager.cc:1428] T bb4b63bee6e24286b6bf7968dc01e0c4 P 19fb8fb17c974ab189c484b2b39240dc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 03:22:15.738243 18424 raft_consensus.cc:482] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:22:15.738452 18424 raft_consensus.cc:504] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.740121 18424 leader_election.cc:288] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19fb8fb17c974ab189c484b2b39240dc (127.10.12.129:34351), d02b12505f3343928b60d401fa75650e (127.10.12.130:33357)
I20260924 03:22:15.748776 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:15.740676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:36331 (local address 127.10.12.130:33357)
0924 03:22:15.740877 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:15.740884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:15.741687 (+ 803us) server_negotiation.cc:407] Connection header received
0924 03:22:15.741967 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:15.741972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:15.742053 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:15.742119 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:15.742967 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.744008 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.746141 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.747471 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.747688 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:15.747766 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:15.748320 (+ 554us) server_negotiation.cc:299] Negotiation successful
0924 03:22:15.748477 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:22:15.749001 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:15.740470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:33357 (local address 127.10.12.131:36331)
0924 03:22:15.741585 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:15.741604 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:15.741804 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:15.742327 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:15.742332 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:15.742372 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:15.742726 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:15.742737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.744180 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.744185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:15.745924 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:15.745941 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.747762 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.747770 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:15.748111 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:15.748174 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 03:22:15.748276 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 03:22:15.748498 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":979,"mutex_wait_us":22,"thread_start_us":1140,"threads_started":1}
I20260924 03:22:15.750129 18206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b63bee6e24286b6bf7968dc01e0c4" candidate_uuid: "e95f7e6500af47b0b152fdb78860783a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d02b12505f3343928b60d401fa75650e" is_pre_election: true
I20260924 03:22:15.750430 18206 raft_consensus.cc:2454] T bb4b63bee6e24286b6bf7968dc01e0c4 P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e95f7e6500af47b0b152fdb78860783a in term 0.
I20260924 03:22:15.751210 18273 leader_election.cc:302] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d02b12505f3343928b60d401fa75650e, e95f7e6500af47b0b152fdb78860783a; no voters:
I20260924 03:22:15.751827 18424 raft_consensus.cc:2781] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:22:15.752007 18424 raft_consensus.cc:482] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:22:15.752122 18424 raft_consensus.cc:3037] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:15.752801 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:15.740527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:34351 (local address 127.10.12.131:36775)
0924 03:22:15.743408 (+ 2881us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:15.743429 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:15.743560 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:15.743904 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:15.743908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:15.743933 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:15.744421 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:15.744433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.746020 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.746026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:15.748096 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:15.748111 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.750147 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.750153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:15.751938 (+ 1785us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:15.752068 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 03:22:15.752221 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 03:22:15.752391 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2725,"mutex_wait_us":79,"thread_start_us":2568,"threads_started":1}
I20260924 03:22:15.752811 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:15.740974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:36775 (local address 127.10.12.129:34351)
0924 03:22:15.741152 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:15.741158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:15.743534 (+ 2376us) server_negotiation.cc:407] Connection header received
0924 03:22:15.743692 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:15.743698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:15.743777 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:15.743875 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:15.744614 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.745807 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.748287 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.749523 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.749682 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:15.749998 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:15.752291 (+ 2293us) server_negotiation.cc:299] Negotiation successful
0924 03:22:15.752458 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 03:22:15.754030 18068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b63bee6e24286b6bf7968dc01e0c4" candidate_uuid: "e95f7e6500af47b0b152fdb78860783a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb8fb17c974ab189c484b2b39240dc" is_pre_election: true
I20260924 03:22:15.754521 18068 raft_consensus.cc:2454] T bb4b63bee6e24286b6bf7968dc01e0c4 P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e95f7e6500af47b0b152fdb78860783a in term 0.
I20260924 03:22:15.755026 18424 raft_consensus.cc:504] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.755659 18424 leader_election.cc:288] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a [CANDIDATE]: Term 1 election: Requested vote from peers 19fb8fb17c974ab189c484b2b39240dc (127.10.12.129:34351), d02b12505f3343928b60d401fa75650e (127.10.12.130:33357)
I20260924 03:22:15.756201 18068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b63bee6e24286b6bf7968dc01e0c4" candidate_uuid: "e95f7e6500af47b0b152fdb78860783a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb8fb17c974ab189c484b2b39240dc"
I20260924 03:22:15.756397 18068 raft_consensus.cc:3037] T bb4b63bee6e24286b6bf7968dc01e0c4 P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:15.756316 18206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b63bee6e24286b6bf7968dc01e0c4" candidate_uuid: "e95f7e6500af47b0b152fdb78860783a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d02b12505f3343928b60d401fa75650e"
I20260924 03:22:15.756531 18206 raft_consensus.cc:3037] T bb4b63bee6e24286b6bf7968dc01e0c4 P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:15.758939 18068 raft_consensus.cc:2454] T bb4b63bee6e24286b6bf7968dc01e0c4 P 19fb8fb17c974ab189c484b2b39240dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e95f7e6500af47b0b152fdb78860783a in term 1.
I20260924 03:22:15.759421 18272 leader_election.cc:302] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19fb8fb17c974ab189c484b2b39240dc, e95f7e6500af47b0b152fdb78860783a; no voters:
I20260924 03:22:15.759727 18424 raft_consensus.cc:2781] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:22:15.759852 18206 raft_consensus.cc:2454] T bb4b63bee6e24286b6bf7968dc01e0c4 P d02b12505f3343928b60d401fa75650e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e95f7e6500af47b0b152fdb78860783a in term 1.
I20260924 03:22:15.766233 18424 raft_consensus.cc:686] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a [term 1 LEADER]: Becoming Leader. State: Replica: e95f7e6500af47b0b152fdb78860783a, State: Running, Role: LEADER
I20260924 03:22:15.766762 18424 consensus_queue.cc:237] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.770509 17905 catalog_manager.cc:5495] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a reported cstate change: term changed from 0 to 1, leader changed from <none> to e95f7e6500af47b0b152fdb78860783a (127.10.12.131). New cstate: current_term: 1 leader_uuid: "e95f7e6500af47b0b152fdb78860783a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } health_report { overall_health: HEALTHY } } }
Sep 24 03:22:15 dist-test-slave-00qs krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220132, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.190@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM
I20260924 03:22:15.826877 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:15.810452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:43939 (local address 127.0.0.1:47744)
0924 03:22:15.811107 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:15.811124 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:15.811256 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:15.811671 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:15.811674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:15.812499 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:15.812786 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:15.812797 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.814247 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.814251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:15.815320 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:15.815333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.816407 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.816412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:15.816537 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:15.817273 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:15.817291 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:15.820072 (+ 2781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:15.824058 (+ 3986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:15.824075 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:15.824082 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:15.824594 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:15.825030 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:15.825033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:15.825036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:15.825258 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:15.825950 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:15.825956 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:15.826138 (+ 182us) client_negotiation.cc:769] Sending connection context
0924 03:22:15.826282 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 03:22:15.826559 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":351,"threads_started":1}
I20260924 03:22:15.827163 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:15.810440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47744 (local address 127.10.12.190:43939)
0924 03:22:15.810660 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:15.810667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:15.811215 (+ 548us) server_negotiation.cc:407] Connection header received
0924 03:22:15.811365 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:15.811377 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:15.811459 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:15.811585 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:15.812958 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.814072 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.815483 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.816160 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.816294 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:15.820297 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:15.820304 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:15.820309 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:15.820351 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:15.823826 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:15.824726 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:15.824732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:15.824736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:15.824884 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:15.825383 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:15.825389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:15.825392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:15.825767 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:15.825921 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:15.826443 (+ 522us) server_negotiation.cc:299] Negotiation successful
0924 03:22:15.826734 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":72}
I20260924 03:22:15.833318 17905 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:47744:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 03:22:15.834625 17905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:22:15.842482 18043 tablet_service.cc:1467] Processing CreateTablet for tablet a0c740aa047048eda8ab906306d8be9f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d8c92e64a3ab411bb650676bc9a1c265]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:22:15.842585 18182 tablet_service.cc:1467] Processing CreateTablet for tablet a0c740aa047048eda8ab906306d8be9f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d8c92e64a3ab411bb650676bc9a1c265]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:22:15.843110 18182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0c740aa047048eda8ab906306d8be9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:15.843133 18043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0c740aa047048eda8ab906306d8be9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:15.848218 18321 tablet_service.cc:1467] Processing CreateTablet for tablet a0c740aa047048eda8ab906306d8be9f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d8c92e64a3ab411bb650676bc9a1c265]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:22:15.848845 18321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0c740aa047048eda8ab906306d8be9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:15.850181 18419 tablet_bootstrap.cc:492] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e: Bootstrap starting.
I20260924 03:22:15.852283 18419 tablet_bootstrap.cc:654] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:15.861140 18417 tablet_bootstrap.cc:492] T a0c740aa047048eda8ab906306d8be9f P e95f7e6500af47b0b152fdb78860783a: Bootstrap starting.
I20260924 03:22:15.861622 18418 tablet_bootstrap.cc:492] T a0c740aa047048eda8ab906306d8be9f P 19fb8fb17c974ab189c484b2b39240dc: Bootstrap starting.
I20260924 03:22:15.863215 18417 tablet_bootstrap.cc:654] T a0c740aa047048eda8ab906306d8be9f P e95f7e6500af47b0b152fdb78860783a: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:15.863487 18418 tablet_bootstrap.cc:654] T a0c740aa047048eda8ab906306d8be9f P 19fb8fb17c974ab189c484b2b39240dc: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:15.874797 18418 tablet_bootstrap.cc:492] T a0c740aa047048eda8ab906306d8be9f P 19fb8fb17c974ab189c484b2b39240dc: No bootstrap required, opened a new log
I20260924 03:22:15.875025 18418 ts_tablet_manager.cc:1397] T a0c740aa047048eda8ab906306d8be9f P 19fb8fb17c974ab189c484b2b39240dc: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20260924 03:22:15.876636 18417 tablet_bootstrap.cc:492] T a0c740aa047048eda8ab906306d8be9f P e95f7e6500af47b0b152fdb78860783a: No bootstrap required, opened a new log
I20260924 03:22:15.876845 18417 ts_tablet_manager.cc:1397] T a0c740aa047048eda8ab906306d8be9f P e95f7e6500af47b0b152fdb78860783a: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.001s
I20260924 03:22:15.877241 18418 raft_consensus.cc:348] T a0c740aa047048eda8ab906306d8be9f P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.877508 18418 raft_consensus.cc:374] T a0c740aa047048eda8ab906306d8be9f P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:15.877583 18418 raft_consensus.cc:729] T a0c740aa047048eda8ab906306d8be9f P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fb8fb17c974ab189c484b2b39240dc, State: Initialized, Role: FOLLOWER
I20260924 03:22:15.877666 18417 raft_consensus.cc:348] T a0c740aa047048eda8ab906306d8be9f P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.877930 18417 raft_consensus.cc:374] T a0c740aa047048eda8ab906306d8be9f P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:15.878017 18417 raft_consensus.cc:729] T a0c740aa047048eda8ab906306d8be9f P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e95f7e6500af47b0b152fdb78860783a, State: Initialized, Role: FOLLOWER
I20260924 03:22:15.877830 18418 consensus_queue.cc:260] T a0c740aa047048eda8ab906306d8be9f P 19fb8fb17c974ab189c484b2b39240dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.878269 18417 consensus_queue.cc:260] T a0c740aa047048eda8ab906306d8be9f P e95f7e6500af47b0b152fdb78860783a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.878422 18428 tablet_replica.cc:439] T a0c740aa047048eda8ab906306d8be9f P 19fb8fb17c974ab189c484b2b39240dc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } } }
I20260924 03:22:15.878780 18428 tablet_replica.cc:442] T a0c740aa047048eda8ab906306d8be9f P 19fb8fb17c974ab189c484b2b39240dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:22:15.878813 18417 ts_tablet_manager.cc:1428] T a0c740aa047048eda8ab906306d8be9f P e95f7e6500af47b0b152fdb78860783a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:22:15.879335 18418 ts_tablet_manager.cc:1428] T a0c740aa047048eda8ab906306d8be9f P 19fb8fb17c974ab189c484b2b39240dc: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260924 03:22:15.881806 18424 tablet_replica.cc:439] T a0c740aa047048eda8ab906306d8be9f P e95f7e6500af47b0b152fdb78860783a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } } }
I20260924 03:22:15.882149 18424 tablet_replica.cc:442] T a0c740aa047048eda8ab906306d8be9f P e95f7e6500af47b0b152fdb78860783a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:22:15.886456 18419 tablet_bootstrap.cc:492] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e: No bootstrap required, opened a new log
I20260924 03:22:15.886667 18419 ts_tablet_manager.cc:1397] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.016s
I20260924 03:22:15.887514 18419 raft_consensus.cc:348] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.887778 18419 raft_consensus.cc:374] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:15.887856 18419 raft_consensus.cc:729] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02b12505f3343928b60d401fa75650e, State: Initialized, Role: FOLLOWER
I20260924 03:22:15.888162 18419 consensus_queue.cc:260] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.888813 18419 ts_tablet_manager.cc:1428] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:22:15.888725 18425 tablet_replica.cc:439] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } } }
I20260924 03:22:15.889153 18425 tablet_replica.cc:442] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:22:15.942713 18425 raft_consensus.cc:482] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:22:15.942981 18425 raft_consensus.cc:504] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.944537 18425 leader_election.cc:288] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19fb8fb17c974ab189c484b2b39240dc (127.10.12.129:34351), e95f7e6500af47b0b152fdb78860783a (127.10.12.131:33897)
I20260924 03:22:15.952804 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:15.945678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:51555 (local address 127.10.12.131:33897)
0924 03:22:15.945850 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:15.945856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:15.945953 (+ 97us) server_negotiation.cc:407] Connection header received
0924 03:22:15.946203 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:15.946208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:15.946290 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:15.946395 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:15.947047 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.947801 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.950512 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.951826 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.952043 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:15.952119 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:15.952413 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 03:22:15.952550 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 03:22:15.952836 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:15.945061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:41955 (local address 127.10.12.129:34351)
0924 03:22:15.945526 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:15.945533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:15.945700 (+ 167us) server_negotiation.cc:407] Connection header received
0924 03:22:15.946035 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:15.946041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:15.946124 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:15.946184 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:15.947025 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.948271 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.950675 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.951647 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.951825 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:15.951905 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:15.952360 (+ 455us) server_negotiation.cc:299] Negotiation successful
0924 03:22:15.952544 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291}
I20260924 03:22:15.952883 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:15.944810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:34351 (local address 127.10.12.130:41955)
0924 03:22:15.945578 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:15.945598 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:15.945828 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:15.946339 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:15.946343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:15.946367 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:15.946791 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:15.946807 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.948400 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.948404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:15.950504 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:15.950514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.951916 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.951922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:15.952151 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:15.952187 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 03:22:15.952290 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 03:22:15.952444 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"spinlock_wait_cycles":13568,"thread_start_us":457,"threads_started":1}
I20260924 03:22:15.953626 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:15.944954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:33897 (local address 127.10.12.130:51555)
0924 03:22:15.945871 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:15.945886 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:15.946041 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:15.946414 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:15.946421 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:15.946443 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:15.946879 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:15.946887 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.947975 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.947981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:15.950272 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:15.950292 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.952091 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.952097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:15.952236 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:15.952263 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 03:22:15.953134 (+ 871us) client_negotiation.cc:240] Negotiation successful
0924 03:22:15.953301 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":865,"threads_started":1}
I20260924 03:22:15.953728 18068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0c740aa047048eda8ab906306d8be9f" candidate_uuid: "d02b12505f3343928b60d401fa75650e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb8fb17c974ab189c484b2b39240dc" is_pre_election: true
I20260924 03:22:15.954020 18068 raft_consensus.cc:2454] T a0c740aa047048eda8ab906306d8be9f P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d02b12505f3343928b60d401fa75650e in term 0.
I20260924 03:22:15.954270 18342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0c740aa047048eda8ab906306d8be9f" candidate_uuid: "d02b12505f3343928b60d401fa75650e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e95f7e6500af47b0b152fdb78860783a" is_pre_election: true
I20260924 03:22:15.954514 18342 raft_consensus.cc:2454] T a0c740aa047048eda8ab906306d8be9f P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d02b12505f3343928b60d401fa75650e in term 0.
I20260924 03:22:15.954545 18133 leader_election.cc:302] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19fb8fb17c974ab189c484b2b39240dc, d02b12505f3343928b60d401fa75650e; no voters:
I20260924 03:22:15.954982 18425 raft_consensus.cc:2781] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:22:15.955113 18425 raft_consensus.cc:482] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:22:15.955206 18425 raft_consensus.cc:3037] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:15.957370 18425 raft_consensus.cc:504] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.957999 18425 leader_election.cc:288] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e [CANDIDATE]: Term 1 election: Requested vote from peers 19fb8fb17c974ab189c484b2b39240dc (127.10.12.129:34351), e95f7e6500af47b0b152fdb78860783a (127.10.12.131:33897)
I20260924 03:22:15.958590 18342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0c740aa047048eda8ab906306d8be9f" candidate_uuid: "d02b12505f3343928b60d401fa75650e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e95f7e6500af47b0b152fdb78860783a"
I20260924 03:22:15.958581 18068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0c740aa047048eda8ab906306d8be9f" candidate_uuid: "d02b12505f3343928b60d401fa75650e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb8fb17c974ab189c484b2b39240dc"
I20260924 03:22:15.958799 18342 raft_consensus.cc:3037] T a0c740aa047048eda8ab906306d8be9f P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:15.958824 18068 raft_consensus.cc:3037] T a0c740aa047048eda8ab906306d8be9f P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:15.961822 18342 raft_consensus.cc:2454] T a0c740aa047048eda8ab906306d8be9f P e95f7e6500af47b0b152fdb78860783a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d02b12505f3343928b60d401fa75650e in term 1.
I20260924 03:22:15.961995 18068 raft_consensus.cc:2454] T a0c740aa047048eda8ab906306d8be9f P 19fb8fb17c974ab189c484b2b39240dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d02b12505f3343928b60d401fa75650e in term 1.
I20260924 03:22:15.962604 18133 leader_election.cc:302] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19fb8fb17c974ab189c484b2b39240dc, d02b12505f3343928b60d401fa75650e; no voters:
I20260924 03:22:15.962927 18425 raft_consensus.cc:2781] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:22:15.963446 18425 raft_consensus.cc:686] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e [term 1 LEADER]: Becoming Leader. State: Replica: d02b12505f3343928b60d401fa75650e, State: Running, Role: LEADER
I20260924 03:22:15.963905 18425 consensus_queue.cc:237] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:15.966020 18461 tablet_replica.cc:439] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e: TxnStatusTablet state changed. Reason: New leader d02b12505f3343928b60d401fa75650e. Latest consensus state: current_term: 1 leader_uuid: "d02b12505f3343928b60d401fa75650e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } } }
I20260924 03:22:15.966548 18461 tablet_replica.cc:442] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:22:15.967325 18463 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 03:22:15.967563 18463 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 03:22:15.967988 17904 catalog_manager.cc:5495] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e reported cstate change: term changed from 0 to 1, leader changed from <none> to d02b12505f3343928b60d401fa75650e (127.10.12.130). New cstate: current_term: 1 leader_uuid: "d02b12505f3343928b60d401fa75650e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } health_report { overall_health: UNKNOWN } } }
I20260924 03:22:16.000669 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:15.994483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:33357 (local address 127.0.0.1:57042)
0924 03:22:15.994846 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:15.994863 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:15.994975 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:15.995588 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:15.995592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:15.995605 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:22:15.995814 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:15.995826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.996981 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.996987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:15.998140 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:15.998154 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.998930 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.998936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:15.999110 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:15.999175 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:22:16.000100 (+ 925us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:22:16.000124 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 03:22:16.000201 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 03:22:16.000373 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220}
I20260924 03:22:16.000646 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:15.994592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57042 (local address 127.10.12.130:33357)
0924 03:22:15.995203 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:15.995207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:15.995225 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:22:15.995319 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:15.995324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:15.995419 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:15.995546 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:22:15.995996 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.996839 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.998285 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:15.998763 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:15.998900 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:15.999370 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:22:15.999945 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:22:16.000121 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:16.000233 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 03:22:16.000320 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509}
I20260924 03:22:16.005504 18342 raft_consensus.cc:1261] T a0c740aa047048eda8ab906306d8be9f P e95f7e6500af47b0b152fdb78860783a [term 1 FOLLOWER]: Refusing update from remote peer d02b12505f3343928b60d401fa75650e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:22:16.005843 18068 raft_consensus.cc:1261] T a0c740aa047048eda8ab906306d8be9f P 19fb8fb17c974ab189c484b2b39240dc [term 1 FOLLOWER]: Refusing update from remote peer d02b12505f3343928b60d401fa75650e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:22:16.006671 18461 consensus_queue.cc:1035] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:22:16.007272 18461 consensus_queue.cc:1035] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:22:16.020104 18428 tablet_replica.cc:439] T a0c740aa047048eda8ab906306d8be9f P 19fb8fb17c974ab189c484b2b39240dc: TxnStatusTablet state changed. Reason: New leader d02b12505f3343928b60d401fa75650e. Latest consensus state: current_term: 1 leader_uuid: "d02b12505f3343928b60d401fa75650e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } } }
I20260924 03:22:16.020488 18428 tablet_replica.cc:442] T a0c740aa047048eda8ab906306d8be9f P 19fb8fb17c974ab189c484b2b39240dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:22:16.021193 18424 tablet_replica.cc:439] T a0c740aa047048eda8ab906306d8be9f P e95f7e6500af47b0b152fdb78860783a: TxnStatusTablet state changed. Reason: New leader d02b12505f3343928b60d401fa75650e. Latest consensus state: current_term: 1 leader_uuid: "d02b12505f3343928b60d401fa75650e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } } }
I20260924 03:22:16.021520 18424 tablet_replica.cc:442] T a0c740aa047048eda8ab906306d8be9f P e95f7e6500af47b0b152fdb78860783a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:22:16.025610 18425 tablet_replica.cc:439] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d02b12505f3343928b60d401fa75650e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } } }
I20260924 03:22:16.025976 18425 tablet_replica.cc:442] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:22:16.026503 18425 tablet_replica.cc:439] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d02b12505f3343928b60d401fa75650e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } } }
I20260924 03:22:16.027889 18425 tablet_replica.cc:442] T a0c740aa047048eda8ab906306d8be9f P d02b12505f3343928b60d401fa75650e: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:22:16.027848 18424 tablet_replica.cc:439] T a0c740aa047048eda8ab906306d8be9f P e95f7e6500af47b0b152fdb78860783a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d02b12505f3343928b60d401fa75650e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } } }
I20260924 03:22:16.028206 18424 tablet_replica.cc:442] T a0c740aa047048eda8ab906306d8be9f P e95f7e6500af47b0b152fdb78860783a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:22:16.029434 18431 mvcc.cc:205] Tried to move back new op lower bound from 7332741677070012416 to 7332741676912328704. Current Snapshot: MvccSnapshot[applied={T|T < 7332741677070012416}]
I20260924 03:22:16.029767 18428 tablet_replica.cc:439] T a0c740aa047048eda8ab906306d8be9f P 19fb8fb17c974ab189c484b2b39240dc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d02b12505f3343928b60d401fa75650e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } } }
I20260924 03:22:16.030012 18428 tablet_replica.cc:442] T a0c740aa047048eda8ab906306d8be9f P 19fb8fb17c974ab189c484b2b39240dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:22:16.027062 18433 mvcc.cc:205] Tried to move back new op lower bound from 7332741677070012416 to 7332741676912328704. Current Snapshot: MvccSnapshot[applied={T|T < 7332741677070012416}]
I20260924 03:22:16.045544 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:16.039233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53942 (local address 127.10.12.131:33897)
0924 03:22:16.039411 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:16.039418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:16.039618 (+ 200us) server_negotiation.cc:407] Connection header received
0924 03:22:16.039873 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:16.039877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:16.040009 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:16.040163 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:22:16.040534 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:16.041284 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:16.043221 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:16.043676 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:16.043793 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:16.044322 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:22:16.044905 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:22:16.045037 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:16.045146 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 03:22:16.045232 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 03:22:16.061851 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:16.056439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:59985 (local address 127.10.12.130:33357)
0924 03:22:16.056633 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:16.056640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:16.056854 (+ 214us) server_negotiation.cc:407] Connection header received
0924 03:22:16.057124 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:16.057127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:16.057192 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:16.057269 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:22:16.057767 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:16.058512 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:16.059932 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:16.060506 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:16.060621 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:16.061079 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:22:16.061206 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:22:16.061279 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:16.061423 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 03:22:16.061510 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:22:16.061918 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:16.056113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:33357 (local address 127.10.12.131:59985)
0924 03:22:16.056749 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:16.056768 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:16.056981 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:16.057299 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:16.057304 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:16.057315 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:22:16.057619 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:16.057629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:16.058647 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:16.058653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:16.059778 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:16.059792 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:16.060691 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:16.060698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:16.060837 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:16.060910 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:22:16.061301 (+ 391us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:22:16.061313 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 03:22:16.061395 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 03:22:16.061549 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":531,"threads_started":1}
I20260924 03:22:16.075271 18206 raft_consensus.cc:1261] T bb4b63bee6e24286b6bf7968dc01e0c4 P d02b12505f3343928b60d401fa75650e [term 1 FOLLOWER]: Refusing update from remote peer e95f7e6500af47b0b152fdb78860783a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:22:16.075546 18068 raft_consensus.cc:1261] T bb4b63bee6e24286b6bf7968dc01e0c4 P 19fb8fb17c974ab189c484b2b39240dc [term 1 FOLLOWER]: Refusing update from remote peer e95f7e6500af47b0b152fdb78860783a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:22:16.076448 18448 consensus_queue.cc:1035] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:22:16.077361 18448 consensus_queue.cc:1035] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:22:16.090513 18432 mvcc.cc:205] Tried to move back new op lower bound from 7332741677355397120 to 7332741676104466432. Current Snapshot: MvccSnapshot[applied={T|T < 7332741677355397120}]
I20260924 03:22:16.132282 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:16.122013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:39967 (local address 127.10.12.131:33897)
0924 03:22:16.122426 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:16.122434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:16.125148 (+ 2714us) server_negotiation.cc:407] Connection header received
0924 03:22:16.125336 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:16.125342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:16.125414 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:16.125487 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:22:16.126364 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:16.127188 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:16.130226 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:16.130699 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:16.130832 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:16.131332 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:22:16.131485 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:22:16.131552 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:16.131840 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 03:22:16.131931 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260924 03:22:16.133714 18475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:16.121305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:33897 (local address 127.10.12.130:39967)
0924 03:22:16.125027 (+ 3722us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:16.125045 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:16.125201 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:16.125529 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:16.125534 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:16.125546 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:22:16.125906 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:16.125923 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:16.127787 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:16.127793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:16.130054 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:16.130072 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:16.130917 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:16.130924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:16.131058 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:16.131124 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:22:16.131696 (+ 572us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:22:16.131706 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 03:22:16.132774 (+ 1068us) client_negotiation.cc:240] Negotiation successful
0924 03:22:16.133284 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3492,"thread_start_us":3322,"threads_started":1}
I20260924 03:22:16.182619 17905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:22:16.182861 17905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:22:16.186246 17905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934: Sending DeleteTablet for 3 replicas of tablet bb4b63bee6e24286b6bf7968dc01e0c4
I20260924 03:22:16.187206 18043 tablet_service.cc:1514] Processing DeleteTablet for tablet bb4b63bee6e24286b6bf7968dc01e0c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:22:16 UTC) from {username='oryx', principal='oryx/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:49630
I20260924 03:22:16.187512 18440 tablet_replica.cc:330] T bb4b63bee6e24286b6bf7968dc01e0c4 P 19fb8fb17c974ab189c484b2b39240dc: stopping tablet replica
I20260924 03:22:16.187465 18182 tablet_service.cc:1514] Processing DeleteTablet for tablet bb4b63bee6e24286b6bf7968dc01e0c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:22:16 UTC) from {username='oryx', principal='oryx/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:57026
I20260924 03:22:16.187484 18321 tablet_service.cc:1514] Processing DeleteTablet for tablet bb4b63bee6e24286b6bf7968dc01e0c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:22:16 UTC) from {username='oryx', principal='oryx/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:53930
I20260924 03:22:16.187717 18440 raft_consensus.cc:2229] T bb4b63bee6e24286b6bf7968dc01e0c4 P 19fb8fb17c974ab189c484b2b39240dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:22:16.187768 18441 tablet_replica.cc:330] T bb4b63bee6e24286b6bf7968dc01e0c4 P d02b12505f3343928b60d401fa75650e: stopping tablet replica
I20260924 03:22:16.187920 18440 raft_consensus.cc:2258] T bb4b63bee6e24286b6bf7968dc01e0c4 P 19fb8fb17c974ab189c484b2b39240dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:22:16.187937 18441 raft_consensus.cc:2229] T bb4b63bee6e24286b6bf7968dc01e0c4 P d02b12505f3343928b60d401fa75650e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:22:16.188151 18441 raft_consensus.cc:2258] T bb4b63bee6e24286b6bf7968dc01e0c4 P d02b12505f3343928b60d401fa75650e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:22:16.188457 18442 tablet_replica.cc:330] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a: stopping tablet replica
I20260924 03:22:16.188647 18442 raft_consensus.cc:2229] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:22:16.189239 18442 raft_consensus.cc:2258] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:22:16.189851 18441 ts_tablet_manager.cc:1905] T bb4b63bee6e24286b6bf7968dc01e0c4 P d02b12505f3343928b60d401fa75650e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:22:16.190594 18442 ts_tablet_manager.cc:1905] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:22:16.193260 18440 ts_tablet_manager.cc:1905] T bb4b63bee6e24286b6bf7968dc01e0c4 P 19fb8fb17c974ab189c484b2b39240dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:22:16.206782 18440 ts_tablet_manager.cc:1918] T bb4b63bee6e24286b6bf7968dc01e0c4 P 19fb8fb17c974ab189c484b2b39240dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:22:16.209218 18441 ts_tablet_manager.cc:1918] T bb4b63bee6e24286b6bf7968dc01e0c4 P d02b12505f3343928b60d401fa75650e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:22:16.209590 18440 log.cc:1196] T bb4b63bee6e24286b6bf7968dc01e0c4 P 19fb8fb17c974ab189c484b2b39240dc: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-0/wal/wals/bb4b63bee6e24286b6bf7968dc01e0c4
I20260924 03:22:16.210151 18440 ts_tablet_manager.cc:1939] T bb4b63bee6e24286b6bf7968dc01e0c4 P 19fb8fb17c974ab189c484b2b39240dc: Deleting consensus metadata
I20260924 03:22:16.210430 18441 log.cc:1196] T bb4b63bee6e24286b6bf7968dc01e0c4 P d02b12505f3343928b60d401fa75650e: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/bb4b63bee6e24286b6bf7968dc01e0c4
I20260924 03:22:16.210970 18441 ts_tablet_manager.cc:1939] T bb4b63bee6e24286b6bf7968dc01e0c4 P d02b12505f3343928b60d401fa75650e: Deleting consensus metadata
I20260924 03:22:16.212080 18442 ts_tablet_manager.cc:1918] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:22:16.212281 17890 catalog_manager.cc:4841] TS 19fb8fb17c974ab189c484b2b39240dc (127.10.12.129:34351): tablet bb4b63bee6e24286b6bf7968dc01e0c4 (table test-table [id=af2f5c5cc0a74460a07b19fdce14cbfd]) successfully deleted
I20260924 03:22:16.213678 17891 catalog_manager.cc:4841] TS d02b12505f3343928b60d401fa75650e (127.10.12.130:33357): tablet bb4b63bee6e24286b6bf7968dc01e0c4 (table test-table [id=af2f5c5cc0a74460a07b19fdce14cbfd]) successfully deleted
I20260924 03:22:16.217289 18442 log.cc:1196] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/bb4b63bee6e24286b6bf7968dc01e0c4
I20260924 03:22:16.217932 18442 ts_tablet_manager.cc:1939] T bb4b63bee6e24286b6bf7968dc01e0c4 P e95f7e6500af47b0b152fdb78860783a: Deleting consensus metadata
I20260924 03:22:16.219403 17892 catalog_manager.cc:4841] TS e95f7e6500af47b0b152fdb78860783a (127.10.12.131:33897): tablet bb4b63bee6e24286b6bf7968dc01e0c4 (table test-table [id=af2f5c5cc0a74460a07b19fdce14cbfd]) successfully deleted
Sep 24 03:22:16 dist-test-slave-00qs krb5kdc[17861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 03:22:16 dist-test-slave-00qs krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:16 dist-test-slave-00qs krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 03:22:16.288148 18280 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:16.273667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47756 (local address 127.10.12.190:43939)
0924 03:22:16.273857 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:16.273864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:16.274376 (+ 512us) server_negotiation.cc:407] Connection header received
0924 03:22:16.274570 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:16.274577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:16.274665 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:16.274777 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:16.276527 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:16.277687 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:16.279866 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:16.280644 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:16.280792 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:16.287842 (+ 7050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47756: BlockingRecv error: recv got EOF from 127.0.0.1:47756 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 03:22:16 dist-test-slave-00qs krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM
I20260924 03:22:16.320366 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:16.295381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47760 (local address 127.10.12.190:43939)
0924 03:22:16.295623 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:16.295630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:16.300633 (+ 5003us) server_negotiation.cc:407] Connection header received
0924 03:22:16.300818 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:16.300824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:16.300943 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:16.301066 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:16.302777 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:16.304014 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:16.306075 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:16.306763 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:16.306925 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:16.312279 (+ 5354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:16.312287 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:16.312292 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:16.312348 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:16.316071 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:16.317148 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:16.317155 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:16.317159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:16.317352 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:16.317779 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:16.317784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:16.317787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:16.318189 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:16.318478 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:16.319598 (+ 1120us) server_negotiation.cc:299] Negotiation successful
0924 03:22:16.319842 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260924 03:22:16.327839 17905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:22:16.328714 17905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:22:16.339620 18321 tablet_service.cc:1467] Processing CreateTablet for tablet e3f12de2984e4a76a25f2284bc876579 (DEFAULT_TABLE table=test-table [id=f327c2a31b344787a43ab3f357703942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:22:16.339646 18182 tablet_service.cc:1467] Processing CreateTablet for tablet e3f12de2984e4a76a25f2284bc876579 (DEFAULT_TABLE table=test-table [id=f327c2a31b344787a43ab3f357703942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:22:16.340143 18321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f12de2984e4a76a25f2284bc876579. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:16.340143 18182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f12de2984e4a76a25f2284bc876579. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:16.344383 18043 tablet_service.cc:1467] Processing CreateTablet for tablet e3f12de2984e4a76a25f2284bc876579 (DEFAULT_TABLE table=test-table [id=f327c2a31b344787a43ab3f357703942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:22:16.344822 18043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f12de2984e4a76a25f2284bc876579. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:16.347416 18419 tablet_bootstrap.cc:492] T e3f12de2984e4a76a25f2284bc876579 P d02b12505f3343928b60d401fa75650e: Bootstrap starting.
I20260924 03:22:16.349089 18419 tablet_bootstrap.cc:654] T e3f12de2984e4a76a25f2284bc876579 P d02b12505f3343928b60d401fa75650e: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:16.353834 18417 tablet_bootstrap.cc:492] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a: Bootstrap starting.
I20260924 03:22:16.355592 18417 tablet_bootstrap.cc:654] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:16.358000 18418 tablet_bootstrap.cc:492] T e3f12de2984e4a76a25f2284bc876579 P 19fb8fb17c974ab189c484b2b39240dc: Bootstrap starting.
I20260924 03:22:16.359911 18418 tablet_bootstrap.cc:654] T e3f12de2984e4a76a25f2284bc876579 P 19fb8fb17c974ab189c484b2b39240dc: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:16.374591 18418 tablet_bootstrap.cc:492] T e3f12de2984e4a76a25f2284bc876579 P 19fb8fb17c974ab189c484b2b39240dc: No bootstrap required, opened a new log
I20260924 03:22:16.375244 18418 ts_tablet_manager.cc:1397] T e3f12de2984e4a76a25f2284bc876579 P 19fb8fb17c974ab189c484b2b39240dc: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.002s
I20260924 03:22:16.377825 18418 raft_consensus.cc:348] T e3f12de2984e4a76a25f2284bc876579 P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:16.378669 18418 raft_consensus.cc:374] T e3f12de2984e4a76a25f2284bc876579 P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:16.378872 18418 raft_consensus.cc:729] T e3f12de2984e4a76a25f2284bc876579 P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fb8fb17c974ab189c484b2b39240dc, State: Initialized, Role: FOLLOWER
I20260924 03:22:16.379312 18418 consensus_queue.cc:260] T e3f12de2984e4a76a25f2284bc876579 P 19fb8fb17c974ab189c484b2b39240dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:16.384846 18418 ts_tablet_manager.cc:1428] T e3f12de2984e4a76a25f2284bc876579 P 19fb8fb17c974ab189c484b2b39240dc: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20260924 03:22:16.393972 18419 tablet_bootstrap.cc:492] T e3f12de2984e4a76a25f2284bc876579 P d02b12505f3343928b60d401fa75650e: No bootstrap required, opened a new log
I20260924 03:22:16.394186 18417 tablet_bootstrap.cc:492] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a: No bootstrap required, opened a new log
I20260924 03:22:16.394335 18419 ts_tablet_manager.cc:1397] T e3f12de2984e4a76a25f2284bc876579 P d02b12505f3343928b60d401fa75650e: Time spent bootstrapping tablet: real 0.047s user 0.015s sys 0.015s
I20260924 03:22:16.394423 18417 ts_tablet_manager.cc:1397] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.010s
I20260924 03:22:16.395386 18419 raft_consensus.cc:348] T e3f12de2984e4a76a25f2284bc876579 P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:16.395790 18419 raft_consensus.cc:374] T e3f12de2984e4a76a25f2284bc876579 P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:16.395973 18419 raft_consensus.cc:729] T e3f12de2984e4a76a25f2284bc876579 P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02b12505f3343928b60d401fa75650e, State: Initialized, Role: FOLLOWER
I20260924 03:22:16.396376 18419 consensus_queue.cc:260] T e3f12de2984e4a76a25f2284bc876579 P d02b12505f3343928b60d401fa75650e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:16.397058 18419 ts_tablet_manager.cc:1428] T e3f12de2984e4a76a25f2284bc876579 P d02b12505f3343928b60d401fa75650e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 03:22:16.395380 18417 raft_consensus.cc:348] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:16.397502 18417 raft_consensus.cc:374] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:16.397624 18417 raft_consensus.cc:729] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e95f7e6500af47b0b152fdb78860783a, State: Initialized, Role: FOLLOWER
I20260924 03:22:16.397965 18417 consensus_queue.cc:260] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:16.399322 18417 ts_tablet_manager.cc:1428] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260924 03:22:16.554283 18448 raft_consensus.cc:482] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:22:16.554553 18448 raft_consensus.cc:504] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:16.555449 18448 leader_election.cc:288] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19fb8fb17c974ab189c484b2b39240dc (127.10.12.129:34351), d02b12505f3343928b60d401fa75650e (127.10.12.130:33357)
I20260924 03:22:16.556219 18206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f12de2984e4a76a25f2284bc876579" candidate_uuid: "e95f7e6500af47b0b152fdb78860783a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d02b12505f3343928b60d401fa75650e" is_pre_election: true
I20260924 03:22:16.556496 18206 raft_consensus.cc:2454] T e3f12de2984e4a76a25f2284bc876579 P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e95f7e6500af47b0b152fdb78860783a in term 0.
I20260924 03:22:16.556442 18067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f12de2984e4a76a25f2284bc876579" candidate_uuid: "e95f7e6500af47b0b152fdb78860783a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb8fb17c974ab189c484b2b39240dc" is_pre_election: true
I20260924 03:22:16.556721 18067 raft_consensus.cc:2454] T e3f12de2984e4a76a25f2284bc876579 P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e95f7e6500af47b0b152fdb78860783a in term 0.
I20260924 03:22:16.557027 18273 leader_election.cc:302] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d02b12505f3343928b60d401fa75650e, e95f7e6500af47b0b152fdb78860783a; no voters:
I20260924 03:22:16.557540 18448 raft_consensus.cc:2781] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:22:16.557679 18448 raft_consensus.cc:482] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:22:16.557789 18448 raft_consensus.cc:3037] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:16.561659 18448 raft_consensus.cc:504] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:16.563174 18206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f12de2984e4a76a25f2284bc876579" candidate_uuid: "e95f7e6500af47b0b152fdb78860783a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d02b12505f3343928b60d401fa75650e"
I20260924 03:22:16.563429 18206 raft_consensus.cc:3037] T e3f12de2984e4a76a25f2284bc876579 P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:16.563335 18067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f12de2984e4a76a25f2284bc876579" candidate_uuid: "e95f7e6500af47b0b152fdb78860783a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb8fb17c974ab189c484b2b39240dc"
I20260924 03:22:16.563575 18067 raft_consensus.cc:3037] T e3f12de2984e4a76a25f2284bc876579 P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:16.566532 18067 raft_consensus.cc:2454] T e3f12de2984e4a76a25f2284bc876579 P 19fb8fb17c974ab189c484b2b39240dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e95f7e6500af47b0b152fdb78860783a in term 1.
I20260924 03:22:16.566533 18206 raft_consensus.cc:2454] T e3f12de2984e4a76a25f2284bc876579 P d02b12505f3343928b60d401fa75650e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e95f7e6500af47b0b152fdb78860783a in term 1.
I20260924 03:22:16.567207 18272 leader_election.cc:302] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 19fb8fb17c974ab189c484b2b39240dc, d02b12505f3343928b60d401fa75650e, e95f7e6500af47b0b152fdb78860783a; no voters:
I20260924 03:22:16.567444 18448 leader_election.cc:288] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a [CANDIDATE]: Term 1 election: Requested vote from peers 19fb8fb17c974ab189c484b2b39240dc (127.10.12.129:34351), d02b12505f3343928b60d401fa75650e (127.10.12.130:33357)
I20260924 03:22:16.567606 18477 raft_consensus.cc:2781] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:22:16.567777 18477 raft_consensus.cc:686] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a [term 1 LEADER]: Becoming Leader. State: Replica: e95f7e6500af47b0b152fdb78860783a, State: Running, Role: LEADER
I20260924 03:22:16.568117 18477 consensus_queue.cc:237] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:16.573938 17905 catalog_manager.cc:5495] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a reported cstate change: term changed from 0 to 1, leader changed from <none> to e95f7e6500af47b0b152fdb78860783a (127.10.12.131). New cstate: current_term: 1 leader_uuid: "e95f7e6500af47b0b152fdb78860783a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } health_report { overall_health: HEALTHY } } }
I20260924 03:22:16.635147 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:16.628188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53944 (local address 127.10.12.131:33897)
0924 03:22:16.628387 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:16.628394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:16.628414 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:22:16.628493 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:16.628498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:16.628585 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:16.628689 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:22:16.629101 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:16.630253 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:16.632341 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:16.633030 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:16.633187 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:16.633811 (+ 624us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:22:16.634032 (+ 221us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:22:16.634137 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:16.634718 (+ 581us) server_negotiation.cc:299] Negotiation successful
0924 03:22:16.634811 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:22:16.638500 18206 raft_consensus.cc:1261] T e3f12de2984e4a76a25f2284bc876579 P d02b12505f3343928b60d401fa75650e [term 1 FOLLOWER]: Refusing update from remote peer e95f7e6500af47b0b152fdb78860783a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:22:16.638491 18067 raft_consensus.cc:1261] T e3f12de2984e4a76a25f2284bc876579 P 19fb8fb17c974ab189c484b2b39240dc [term 1 FOLLOWER]: Refusing update from remote peer e95f7e6500af47b0b152fdb78860783a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:22:16.639271 18448 consensus_queue.cc:1035] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:22:16.639595 18424 consensus_queue.cc:1035] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:22:16.671983 17905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:22:16.672343 17905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:22:16.677101 17905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934: Sending DeleteTablet for 3 replicas of tablet e3f12de2984e4a76a25f2284bc876579
I20260924 03:22:16.678134 18043 tablet_service.cc:1514] Processing DeleteTablet for tablet e3f12de2984e4a76a25f2284bc876579 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:22:16 UTC) from {username='oryx', principal='oryx/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:49630
I20260924 03:22:16.678457 18440 tablet_replica.cc:330] T e3f12de2984e4a76a25f2284bc876579 P 19fb8fb17c974ab189c484b2b39240dc: stopping tablet replica
I20260924 03:22:16.678459 18182 tablet_service.cc:1514] Processing DeleteTablet for tablet e3f12de2984e4a76a25f2284bc876579 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:22:16 UTC) from {username='oryx', principal='oryx/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:57026
I20260924 03:22:16.678761 18441 tablet_replica.cc:330] T e3f12de2984e4a76a25f2284bc876579 P d02b12505f3343928b60d401fa75650e: stopping tablet replica
I20260924 03:22:16.678929 18441 raft_consensus.cc:2229] T e3f12de2984e4a76a25f2284bc876579 P d02b12505f3343928b60d401fa75650e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:22:16.679191 18441 raft_consensus.cc:2258] T e3f12de2984e4a76a25f2284bc876579 P d02b12505f3343928b60d401fa75650e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:22:16.680119 18321 tablet_service.cc:1514] Processing DeleteTablet for tablet e3f12de2984e4a76a25f2284bc876579 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:22:16 UTC) from {username='oryx', principal='oryx/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:53930
I20260924 03:22:16.680466 18442 tablet_replica.cc:330] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a: stopping tablet replica
I20260924 03:22:16.680657 18442 raft_consensus.cc:2229] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:22:16.681051 18442 raft_consensus.cc:2258] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:22:16.681684 18441 ts_tablet_manager.cc:1905] T e3f12de2984e4a76a25f2284bc876579 P d02b12505f3343928b60d401fa75650e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:22:16.682466 17904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:22:16.683178 17904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:22:16.684863 18321 tablet_service.cc:1514] Processing DeleteTablet for tablet e3f12de2984e4a76a25f2284bc876579 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:22:16 UTC) from {username='oryx', principal='oryx/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:53930
I20260924 03:22:16.685623 18182 tablet_service.cc:1514] Processing DeleteTablet for tablet e3f12de2984e4a76a25f2284bc876579 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:22:16 UTC) from {username='oryx', principal='oryx/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:57026
I20260924 03:22:16.690663 18440 raft_consensus.cc:2229] T e3f12de2984e4a76a25f2284bc876579 P 19fb8fb17c974ab189c484b2b39240dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:22:16.691300 18440 pending_rounds.cc:62] T e3f12de2984e4a76a25f2284bc876579 P 19fb8fb17c974ab189c484b2b39240dc: Trying to abort 2 pending ops.
I20260924 03:22:16.691409 18440 pending_rounds.cc:68] T e3f12de2984e4a76a25f2284bc876579 P 19fb8fb17c974ab189c484b2b39240dc: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7332741679664087040 op_type: WRITE_OP write_request { tablet_id: "e3f12de2984e4a76a25f2284bc876579" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\224\261\322\325\006\"8\n\ntest-admin\022*\n f327c2a31b344787a43ab3f357703942\020\001\030\001 \001(\001" signature: "<redacted>""\211\346\005\006\353J\330u\342\364\207\212\351G\325x\236\222\006\315\334\274\252&\331\355}\212\001\267\343_\245\366\230AeX1M\277\242W\007\254\376\203#\345\315#/!R\315=c\256\321\326\367\234\010\201" signing_key_seq_num: 0 } } request_id { client_id: "52e66e05824d490db78f6a0d3df7fb14" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260924 03:22:16.692137 18440 pending_rounds.cc:68] T e3f12de2984e4a76a25f2284bc876579 P 19fb8fb17c974ab189c484b2b39240dc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332741679385952256 op_type: NO_OP noop_request { }
I20260924 03:22:16.692277 18440 raft_consensus.cc:2866] T e3f12de2984e4a76a25f2284bc876579 P 19fb8fb17c974ab189c484b2b39240dc [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260924 03:22:16.692410 18440 raft_consensus.cc:2258] T e3f12de2984e4a76a25f2284bc876579 P 19fb8fb17c974ab189c484b2b39240dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:22:16.692493 18321 tablet_service.cc:1467] Processing CreateTablet for tablet 2a758b61526e4d098d1a77123a781999 (DEFAULT_TABLE table=test-table [id=48b4f038f6bd445db21fbab6a8559161]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:22:16.693022 18321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a758b61526e4d098d1a77123a781999. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:16.693542 18043 tablet_service.cc:1467] Processing CreateTablet for tablet 2a758b61526e4d098d1a77123a781999 (DEFAULT_TABLE table=test-table [id=48b4f038f6bd445db21fbab6a8559161]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:22:16.693953 18043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a758b61526e4d098d1a77123a781999. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:16.698004 18182 tablet_service.cc:1467] Processing CreateTablet for tablet 2a758b61526e4d098d1a77123a781999 (DEFAULT_TABLE table=test-table [id=48b4f038f6bd445db21fbab6a8559161]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:22:16.698578 18182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a758b61526e4d098d1a77123a781999. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:16.706700 18418 tablet_bootstrap.cc:492] T 2a758b61526e4d098d1a77123a781999 P 19fb8fb17c974ab189c484b2b39240dc: Bootstrap starting.
I20260924 03:22:16.708426 18418 tablet_bootstrap.cc:654] T 2a758b61526e4d098d1a77123a781999 P 19fb8fb17c974ab189c484b2b39240dc: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:16.712137 18441 ts_tablet_manager.cc:1918] T e3f12de2984e4a76a25f2284bc876579 P d02b12505f3343928b60d401fa75650e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:22:16.713616 18419 tablet_bootstrap.cc:492] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e: Bootstrap starting.
I20260924 03:22:16.715536 18419 tablet_bootstrap.cc:654] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:16.719376 18441 log.cc:1196] T e3f12de2984e4a76a25f2284bc876579 P d02b12505f3343928b60d401fa75650e: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/e3f12de2984e4a76a25f2284bc876579
I20260924 03:22:16.720556 18441 ts_tablet_manager.cc:1939] T e3f12de2984e4a76a25f2284bc876579 P d02b12505f3343928b60d401fa75650e: Deleting consensus metadata
I20260924 03:22:16.721192 18417 tablet_bootstrap.cc:492] T 2a758b61526e4d098d1a77123a781999 P e95f7e6500af47b0b152fdb78860783a: Bootstrap starting.
W20260924 03:22:16.722618 18497 consensus_queue.cc:1159] T e3f12de2984e4a76a25f2284bc876579 P 19fb8fb17c974ab189c484b2b39240dc [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "19fb8fb17c974ab189c484b2b39240dc" status { last_received { term: 1 index: 2 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 2 } }
I20260924 03:22:16.723235 18417 tablet_bootstrap.cc:654] T 2a758b61526e4d098d1a77123a781999 P e95f7e6500af47b0b152fdb78860783a: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:16.723706 18418 tablet_bootstrap.cc:492] T 2a758b61526e4d098d1a77123a781999 P 19fb8fb17c974ab189c484b2b39240dc: No bootstrap required, opened a new log
I20260924 03:22:16.723997 18418 ts_tablet_manager.cc:1397] T 2a758b61526e4d098d1a77123a781999 P 19fb8fb17c974ab189c484b2b39240dc: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.003s
I20260924 03:22:16.726507 18418 raft_consensus.cc:348] T 2a758b61526e4d098d1a77123a781999 P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:16.726958 18418 raft_consensus.cc:374] T 2a758b61526e4d098d1a77123a781999 P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:16.727080 18418 raft_consensus.cc:729] T 2a758b61526e4d098d1a77123a781999 P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fb8fb17c974ab189c484b2b39240dc, State: Initialized, Role: FOLLOWER
I20260924 03:22:16.727428 18418 consensus_queue.cc:260] T 2a758b61526e4d098d1a77123a781999 P 19fb8fb17c974ab189c484b2b39240dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:16.732151 18418 ts_tablet_manager.cc:1428] T 2a758b61526e4d098d1a77123a781999 P 19fb8fb17c974ab189c484b2b39240dc: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20260924 03:22:16.732774 18440 ts_tablet_manager.cc:1905] T e3f12de2984e4a76a25f2284bc876579 P 19fb8fb17c974ab189c484b2b39240dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:22:16.736096 18442 ts_tablet_manager.cc:1905] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:22:16.742333 18440 ts_tablet_manager.cc:1918] T e3f12de2984e4a76a25f2284bc876579 P 19fb8fb17c974ab189c484b2b39240dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:22:16.743436 18442 ts_tablet_manager.cc:1918] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:22:16.743685 18442 log.cc:1196] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/e3f12de2984e4a76a25f2284bc876579
I20260924 03:22:16.744720 18442 ts_tablet_manager.cc:1939] T e3f12de2984e4a76a25f2284bc876579 P e95f7e6500af47b0b152fdb78860783a: Deleting consensus metadata
I20260924 03:22:16.745482 18440 log.cc:1196] T e3f12de2984e4a76a25f2284bc876579 P 19fb8fb17c974ab189c484b2b39240dc: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-0/wal/wals/e3f12de2984e4a76a25f2284bc876579
I20260924 03:22:16.746022 18440 ts_tablet_manager.cc:1939] T e3f12de2984e4a76a25f2284bc876579 P 19fb8fb17c974ab189c484b2b39240dc: Deleting consensus metadata
I20260924 03:22:16.746111 17892 catalog_manager.cc:4841] TS e95f7e6500af47b0b152fdb78860783a (127.10.12.131:33897): tablet e3f12de2984e4a76a25f2284bc876579 (table test-table [id=f327c2a31b344787a43ab3f357703942]) successfully deleted
W20260924 03:22:16.747186 17892 catalog_manager.cc:4808] TS e95f7e6500af47b0b152fdb78860783a (127.10.12.131:33897): delete failed for tablet e3f12de2984e4a76a25f2284bc876579 because the tablet was not found. No further retry: Not found: Tablet not found: e3f12de2984e4a76a25f2284bc876579
I20260924 03:22:16.748667 17890 catalog_manager.cc:4841] TS 19fb8fb17c974ab189c484b2b39240dc (127.10.12.129:34351): tablet e3f12de2984e4a76a25f2284bc876579 (table test-table [id=f327c2a31b344787a43ab3f357703942]) successfully deleted
I20260924 03:22:16.749130 17891 catalog_manager.cc:4841] TS d02b12505f3343928b60d401fa75650e (127.10.12.130:33357): tablet e3f12de2984e4a76a25f2284bc876579 (table test-table [id=f327c2a31b344787a43ab3f357703942]) successfully deleted
W20260924 03:22:16.750227 17891 catalog_manager.cc:4808] TS d02b12505f3343928b60d401fa75650e (127.10.12.130:33357): delete failed for tablet e3f12de2984e4a76a25f2284bc876579 because the tablet was not found. No further retry: Not found: Tablet not found: e3f12de2984e4a76a25f2284bc876579
I20260924 03:22:16.752934 18417 tablet_bootstrap.cc:492] T 2a758b61526e4d098d1a77123a781999 P e95f7e6500af47b0b152fdb78860783a: No bootstrap required, opened a new log
I20260924 03:22:16.753131 18417 ts_tablet_manager.cc:1397] T 2a758b61526e4d098d1a77123a781999 P e95f7e6500af47b0b152fdb78860783a: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.001s
I20260924 03:22:16.753847 18417 raft_consensus.cc:348] T 2a758b61526e4d098d1a77123a781999 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:16.754092 18417 raft_consensus.cc:374] T 2a758b61526e4d098d1a77123a781999 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:16.754179 18417 raft_consensus.cc:729] T 2a758b61526e4d098d1a77123a781999 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e95f7e6500af47b0b152fdb78860783a, State: Initialized, Role: FOLLOWER
I20260924 03:22:16.754482 18417 consensus_queue.cc:260] T 2a758b61526e4d098d1a77123a781999 P e95f7e6500af47b0b152fdb78860783a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:16.755019 18417 ts_tablet_manager.cc:1428] T 2a758b61526e4d098d1a77123a781999 P e95f7e6500af47b0b152fdb78860783a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 03:22:16.756264 18419 tablet_bootstrap.cc:492] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e: No bootstrap required, opened a new log
I20260924 03:22:16.756508 18419 ts_tablet_manager.cc:1397] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.008s
I20260924 03:22:16.757359 18419 raft_consensus.cc:348] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:16.757678 18419 raft_consensus.cc:374] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:16.757799 18419 raft_consensus.cc:729] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02b12505f3343928b60d401fa75650e, State: Initialized, Role: FOLLOWER
I20260924 03:22:16.758144 18419 consensus_queue.cc:260] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:16.758750 18419 ts_tablet_manager.cc:1428] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:22:16.906273 18471 raft_consensus.cc:482] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:22:16.906558 18471 raft_consensus.cc:504] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:16.907377 18471 leader_election.cc:288] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19fb8fb17c974ab189c484b2b39240dc (127.10.12.129:34351), e95f7e6500af47b0b152fdb78860783a (127.10.12.131:33897)
I20260924 03:22:16.907992 18342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a758b61526e4d098d1a77123a781999" candidate_uuid: "d02b12505f3343928b60d401fa75650e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e95f7e6500af47b0b152fdb78860783a" is_pre_election: true
I20260924 03:22:16.908010 18067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a758b61526e4d098d1a77123a781999" candidate_uuid: "d02b12505f3343928b60d401fa75650e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb8fb17c974ab189c484b2b39240dc" is_pre_election: true
I20260924 03:22:16.908254 18342 raft_consensus.cc:2454] T 2a758b61526e4d098d1a77123a781999 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d02b12505f3343928b60d401fa75650e in term 0.
I20260924 03:22:16.908258 18067 raft_consensus.cc:2454] T 2a758b61526e4d098d1a77123a781999 P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d02b12505f3343928b60d401fa75650e in term 0.
I20260924 03:22:16.908829 18135 leader_election.cc:302] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d02b12505f3343928b60d401fa75650e, e95f7e6500af47b0b152fdb78860783a; no voters:
I20260924 03:22:16.909176 18471 raft_consensus.cc:2781] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:22:16.909307 18471 raft_consensus.cc:482] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:22:16.909413 18471 raft_consensus.cc:3037] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:16.911559 18471 raft_consensus.cc:504] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:16.912246 18471 leader_election.cc:288] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e [CANDIDATE]: Term 1 election: Requested vote from peers 19fb8fb17c974ab189c484b2b39240dc (127.10.12.129:34351), e95f7e6500af47b0b152fdb78860783a (127.10.12.131:33897)
I20260924 03:22:16.912791 18067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a758b61526e4d098d1a77123a781999" candidate_uuid: "d02b12505f3343928b60d401fa75650e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb8fb17c974ab189c484b2b39240dc"
I20260924 03:22:16.912915 18342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a758b61526e4d098d1a77123a781999" candidate_uuid: "d02b12505f3343928b60d401fa75650e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e95f7e6500af47b0b152fdb78860783a"
I20260924 03:22:16.913033 18067 raft_consensus.cc:3037] T 2a758b61526e4d098d1a77123a781999 P 19fb8fb17c974ab189c484b2b39240dc [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:16.913136 18342 raft_consensus.cc:3037] T 2a758b61526e4d098d1a77123a781999 P e95f7e6500af47b0b152fdb78860783a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:16.915985 18067 raft_consensus.cc:2454] T 2a758b61526e4d098d1a77123a781999 P 19fb8fb17c974ab189c484b2b39240dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d02b12505f3343928b60d401fa75650e in term 1.
I20260924 03:22:16.915992 18342 raft_consensus.cc:2454] T 2a758b61526e4d098d1a77123a781999 P e95f7e6500af47b0b152fdb78860783a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d02b12505f3343928b60d401fa75650e in term 1.
I20260924 03:22:16.916553 18133 leader_election.cc:302] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 19fb8fb17c974ab189c484b2b39240dc, d02b12505f3343928b60d401fa75650e, e95f7e6500af47b0b152fdb78860783a; no voters:
I20260924 03:22:16.916977 18471 raft_consensus.cc:2781] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:22:16.917405 18471 raft_consensus.cc:686] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e [term 1 LEADER]: Becoming Leader. State: Replica: d02b12505f3343928b60d401fa75650e, State: Running, Role: LEADER
I20260924 03:22:16.917910 18471 consensus_queue.cc:237] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } }
I20260924 03:22:16.920807 17905 catalog_manager.cc:5495] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e reported cstate change: term changed from 0 to 1, leader changed from <none> to d02b12505f3343928b60d401fa75650e (127.10.12.130). New cstate: current_term: 1 leader_uuid: "d02b12505f3343928b60d401fa75650e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d02b12505f3343928b60d401fa75650e" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 33357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 } health_report { overall_health: UNKNOWN } } }
I20260924 03:22:16.967864 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:16.962662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57056 (local address 127.10.12.130:33357)
0924 03:22:16.963266 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:16.963271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:16.963288 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:22:16.963345 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:16.963349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:16.963412 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:16.963502 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:22:16.963915 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:16.964766 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:16.966142 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:16.966564 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:16.966678 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:16.967123 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:22:16.967241 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:22:16.967315 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:16.967465 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 03:22:16.967550 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":360,"threads_started":1}
I20260924 03:22:16.982155 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:16.976691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:41899 (local address 127.10.12.130:33357)
0924 03:22:16.976875 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:16.976886 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:16.977307 (+ 421us) server_negotiation.cc:407] Connection header received
0924 03:22:16.977468 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:16.977472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:16.977541 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:16.977620 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:22:16.978115 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:16.979160 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:16.980695 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:16.981094 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:16.981198 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:16.981547 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:22:16.981646 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:22:16.981704 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:16.981852 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 03:22:16.981916 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 03:22:16.982267 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:16.976640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:33357 (local address 127.10.12.130:41899)
0924 03:22:16.977209 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:16.977227 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:16.977363 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:16.977733 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:16.977739 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:16.977752 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:22:16.977980 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:16.977988 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:16.979385 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:16.979389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:16.980555 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:16.980571 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:16.981249 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:16.981253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:16.981373 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:16.981412 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:22:16.981746 (+ 334us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:22:16.981758 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 03:22:16.981834 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 03:22:16.981953 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":1,"thread_start_us":322,"threads_started":1}
I20260924 03:22:16.990779 18342 raft_consensus.cc:1261] T 2a758b61526e4d098d1a77123a781999 P e95f7e6500af47b0b152fdb78860783a [term 1 FOLLOWER]: Refusing update from remote peer d02b12505f3343928b60d401fa75650e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:22:16.990845 18067 raft_consensus.cc:1261] T 2a758b61526e4d098d1a77123a781999 P 19fb8fb17c974ab189c484b2b39240dc [term 1 FOLLOWER]: Refusing update from remote peer d02b12505f3343928b60d401fa75650e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:22:16.991387 18502 consensus_queue.cc:1035] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb8fb17c974ab189c484b2b39240dc" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:22:16.991695 18471 consensus_queue.cc:1035] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e95f7e6500af47b0b152fdb78860783a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 33897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:22:17.046025 17905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:22:17.046299 17905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:22:17.049443 17905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 809ce94c916f428f977b1e0c1a468934: Sending DeleteTablet for 3 replicas of tablet 2a758b61526e4d098d1a77123a781999
I20260924 03:22:17.050379 18043 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a758b61526e4d098d1a77123a781999 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:22:17 UTC) from {username='oryx', principal='oryx/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:49630
I20260924 03:22:17.050421 18321 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a758b61526e4d098d1a77123a781999 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:22:17 UTC) from {username='oryx', principal='oryx/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:53930
I20260924 03:22:17.050678 18440 tablet_replica.cc:330] T 2a758b61526e4d098d1a77123a781999 P 19fb8fb17c974ab189c484b2b39240dc: stopping tablet replica
I20260924 03:22:17.050678 18442 tablet_replica.cc:330] T 2a758b61526e4d098d1a77123a781999 P e95f7e6500af47b0b152fdb78860783a: stopping tablet replica
I20260924 03:22:17.050891 18442 raft_consensus.cc:2229] T 2a758b61526e4d098d1a77123a781999 P e95f7e6500af47b0b152fdb78860783a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:22:17.050891 18440 raft_consensus.cc:2229] T 2a758b61526e4d098d1a77123a781999 P 19fb8fb17c974ab189c484b2b39240dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:22:17.050956 18181 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a758b61526e4d098d1a77123a781999 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:22:17 UTC) from {username='oryx', principal='oryx/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:57026
I20260924 03:22:17.051084 18442 raft_consensus.cc:2258] T 2a758b61526e4d098d1a77123a781999 P e95f7e6500af47b0b152fdb78860783a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:22:17.051084 18440 raft_consensus.cc:2258] T 2a758b61526e4d098d1a77123a781999 P 19fb8fb17c974ab189c484b2b39240dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:22:17.051360 18441 tablet_replica.cc:330] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e: stopping tablet replica
I20260924 03:22:17.051579 18441 raft_consensus.cc:2229] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:22:17.052183 18441 raft_consensus.cc:2258] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:22:17.052367 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 17977
I20260924 03:22:17.052752 18442 ts_tablet_manager.cc:1905] T 2a758b61526e4d098d1a77123a781999 P e95f7e6500af47b0b152fdb78860783a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:22:17.054844 18441 ts_tablet_manager.cc:1905] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:22:17.059648 18442 ts_tablet_manager.cc:1918] T 2a758b61526e4d098d1a77123a781999 P e95f7e6500af47b0b152fdb78860783a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:22:17.059819 18442 log.cc:1196] T 2a758b61526e4d098d1a77123a781999 P e95f7e6500af47b0b152fdb78860783a: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/2a758b61526e4d098d1a77123a781999
I20260924 03:22:17.060238 18442 ts_tablet_manager.cc:1939] T 2a758b61526e4d098d1a77123a781999 P e95f7e6500af47b0b152fdb78860783a: Deleting consensus metadata
I20260924 03:22:17.061482 17892 catalog_manager.cc:4841] TS e95f7e6500af47b0b152fdb78860783a (127.10.12.131:33897): tablet 2a758b61526e4d098d1a77123a781999 (table test-table [id=48b4f038f6bd445db21fbab6a8559161]) successfully deleted
I20260924 03:22:17.062944 18441 ts_tablet_manager.cc:1918] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:22:17.063107 18441 log.cc:1196] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/2a758b61526e4d098d1a77123a781999
I20260924 03:22:17.063442 18441 ts_tablet_manager.cc:1939] T 2a758b61526e4d098d1a77123a781999 P d02b12505f3343928b60d401fa75650e: Deleting consensus metadata
I20260924 03:22:17.064525 18440 ts_tablet_manager.cc:1905] T 2a758b61526e4d098d1a77123a781999 P 19fb8fb17c974ab189c484b2b39240dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:22:17.064738 17891 catalog_manager.cc:4841] TS d02b12505f3343928b60d401fa75650e (127.10.12.130:33357): tablet 2a758b61526e4d098d1a77123a781999 (table test-table [id=48b4f038f6bd445db21fbab6a8559161]) successfully deleted
I20260924 03:22:17.070618 18440 ts_tablet_manager.cc:1918] T 2a758b61526e4d098d1a77123a781999 P 19fb8fb17c974ab189c484b2b39240dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:22:17.070881 18440 log.cc:1196] T 2a758b61526e4d098d1a77123a781999 P 19fb8fb17c974ab189c484b2b39240dc: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790220067268388-10290-0/minicluster-data/ts-0/wal/wals/2a758b61526e4d098d1a77123a781999
I20260924 03:22:17.071218 18440 ts_tablet_manager.cc:1939] T 2a758b61526e4d098d1a77123a781999 P 19fb8fb17c974ab189c484b2b39240dc: Deleting consensus metadata
I20260924 03:22:17.072837 17890 catalog_manager.cc:4841] TS 19fb8fb17c974ab189c484b2b39240dc (127.10.12.129:34351): tablet 2a758b61526e4d098d1a77123a781999 (table test-table [id=48b4f038f6bd445db21fbab6a8559161]) successfully deleted
Sep 24 03:22:17 dist-test-slave-00qs krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.12.129@KRBTEST.COM
I20260924 03:22:17.085577 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:17.056199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49654 (local address 127.10.12.129:34351)
0924 03:22:17.067390 (+ 11191us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:17.067396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:17.067420 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:22:17.067485 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:17.067490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:17.067602 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:17.067739 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:17.069516 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:17.070578 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:17.072586 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:17.073211 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:17.073319 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:17.078033 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:17.078057 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:17.078080 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:17.078131 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:17.082089 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:17.083077 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:17.083088 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:17.083091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:17.083260 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:17.083688 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:17.083693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:17.083696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:17.084263 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:17.084668 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:17.084884 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 03:22:17.085183 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11051,"thread_start_us":10967,"threads_started":1}
I20260924 03:22:17.086614 18108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:17.263978 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 17977
I20260924 03:22:17.291312 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 18117
Sep 24 03:22:17 dist-test-slave-00qs krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.12.130@KRBTEST.COM
I20260924 03:22:17.309888 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:17.292376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57058 (local address 127.10.12.130:33357)
0924 03:22:17.292547 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:17.292551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:17.292568 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:22:17.292758 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:17.292763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:17.292832 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:17.292919 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:17.294563 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:17.295402 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:17.297480 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:17.297998 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:17.298114 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:17.303366 (+ 5252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:17.303383 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:17.303401 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:17.303491 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:17.306746 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:17.307718 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:17.307727 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:17.307730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:17.307919 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:17.308353 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:17.308357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:17.308359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:17.308727 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:17.308982 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:17.309351 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 03:22:17.309578 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 03:22:17.310557 18247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:17.457401 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 18117
I20260924 03:22:17.476588 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 18256
Sep 24 03:22:17 dist-test-slave-00qs krb5kdc[17861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.12.131@KRBTEST.COM
I20260924 03:22:17.492952 18519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:17.477346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53958 (local address 127.10.12.131:33897)
0924 03:22:17.477868 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:17.477873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:17.477889 (+ 16us) server_negotiation.cc:407] Connection header received
0924 03:22:17.477948 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:17.477951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:17.478009 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:17.478107 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:17.479491 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:17.480350 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:17.481574 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:17.481968 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:17.482080 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:17.486726 (+ 4646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:17.486744 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:17.486762 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:17.486808 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:17.489743 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:17.490616 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:17.490625 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:17.490627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:17.490772 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:17.491226 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:17.491232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:17.491235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:17.491642 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:17.491892 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:17.492356 (+ 464us) server_negotiation.cc:299] Negotiation successful
0924 03:22:17.492550 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":301,"threads_started":1}
I20260924 03:22:17.493695 18387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:17.617815 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 18256
I20260924 03:22:17.635241 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 17874
I20260924 03:22:17.636372 17946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:17.754153 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 17874
2026-09-24T03:22:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6828 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:22:17 dist-test-slave-00qs krb5kdc[18524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:22:17 dist-test-slave-00qs krb5kdc[18524](info): set up 2 sockets
Sep 24 03:22:17 dist-test-slave-00qs krb5kdc[18524](info): commencing operation
krb5kdc: starting...
W20260924 03:22:18.796965 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:22:18 dist-test-slave-00qs krb5kdc[18524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:22:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:22:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:22:19.024762 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:43601
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:43601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:22:19.359494 18537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:19.359838 18537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:19.359939 18537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:19.367707 18537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:22:19.367848 18537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:19.367906 18537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:19.368011 18537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:22:19.368073 18537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:22:19.378389 18537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:43601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:43601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:19.379778 18537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:19.381145 18537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:19.393617 18545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:19.393779 18542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:19.394202 18543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:19.394536 18537 server_base.cc:1034] running on GCE node
I20260924 03:22:19.395373 18537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:19.397486 18537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:19.398819 18537 hybrid_clock.cc:648] HybridClock initialized: now 1790220139398682 us; error 120 us; skew 500 ppm
I20260924 03:22:19.399289 18537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T03:22:19Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1722 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:22:19 dist-test-slave-00qs krb5kdc[18551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:22:19 dist-test-slave-00qs krb5kdc[18551](info): set up 2 sockets
Sep 24 03:22:19 dist-test-slave-00qs krb5kdc[18551](info): commencing operation
krb5kdc: starting...
W20260924 03:22:20.513022 10290 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:22:20 dist-test-slave-00qs krb5kdc[18551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:22:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:22:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:22:20.744948 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:36271
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:43807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:36271
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:22:21.087397 18564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:21.087731 18564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:21.087857 18564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:21.095472 18564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:22:21.095623 18564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:21.095696 18564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:21.095772 18564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:22:21.095831 18564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:22:21.106177 18564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:43807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:36271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:36271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:21.107612 18564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:21.109019 18564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:21.122166 18569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:21.122110 18572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:21.122071 18570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:21.122856 18564 server_base.cc:1034] running on GCE node
I20260924 03:22:21.123662 18564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:21.125264 18564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:21.126513 18564 hybrid_clock.cc:648] HybridClock initialized: now 1790220141126445 us; error 43 us; skew 500 ppm
I20260924 03:22:21.126889 18564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:21 dist-test-slave-00qs krb5kdc[18551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220141, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:21.138481 18564 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:22:21.140378 18564 webserver.cc:460] Webserver started at http://127.10.12.190:46455/ using document root <none> and password file <none>
I20260924 03:22:21.141213 18564 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:21.141338 18564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:21.141678 18564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:21.145067 18564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "b69ccdc0e31f41978c85f75a574a9cdc"
format_stamp: "Formatted at 2026-09-24 03:22:21 on dist-test-slave-00qs"
server_key: "cc4db452e170dc98d79bb83160f84b66"
server_key_iv: "72dfd60238890f27ad33f39bda0af738"
server_key_version: "encryptionkey@0"
I20260924 03:22:21.145928 18564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "b69ccdc0e31f41978c85f75a574a9cdc"
format_stamp: "Formatted at 2026-09-24 03:22:21 on dist-test-slave-00qs"
server_key: "cc4db452e170dc98d79bb83160f84b66"
server_key_iv: "72dfd60238890f27ad33f39bda0af738"
server_key_version: "encryptionkey@0"
I20260924 03:22:21.151922 18564 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260924 03:22:21.156425 18579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:21.158094 18564 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260924 03:22:21.158298 18564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "b69ccdc0e31f41978c85f75a574a9cdc"
format_stamp: "Formatted at 2026-09-24 03:22:21 on dist-test-slave-00qs"
server_key: "cc4db452e170dc98d79bb83160f84b66"
server_key_iv: "72dfd60238890f27ad33f39bda0af738"
server_key_version: "encryptionkey@0"
I20260924 03:22:21.158514 18564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:21.187084 18564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:21.191712 18564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:21.192046 18564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:21.210230 18564 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:36271
I20260924 03:22:21.210253 18630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:36271 every 8 connection(s)
I20260924 03:22:21.211676 18564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:22:21.215639 18631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:21.219518 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 18564
I20260924 03:22:21.219734 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:22:21.220563 10290 external_mini_cluster.cc:1428] Setting key e6679e78cb5af6b2fdb1921b4ad2614c
I20260924 03:22:21.229861 18631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc: Bootstrap starting.
I20260924 03:22:21.234141 18631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:21.235612 18631 log.cc:824] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc: Log is configured to *not* fsync() on all Append() calls
Sep 24 03:22:21 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:22:21.238922 18631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc: No bootstrap required, opened a new log
I20260924 03:22:21.245085 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:21.222814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36196 (local address 127.10.12.190:36271)
0924 03:22:21.223583 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:21.223597 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:21.223640 (+ 43us) server_negotiation.cc:407] Connection header received
0924 03:22:21.225342 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:21.225369 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:21.225872 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:21.226236 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:21.228267 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:21.229702 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:21.231540 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:21.232397 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:21.232548 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:21.237590 (+ 5042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:21.237599 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:21.237617 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:21.237656 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:21.240709 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:21.241492 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:21.241499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:21.241507 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:21.241704 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:21.242101 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:21.242104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:21.242106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:21.242678 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:21.243001 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:21.243474 (+ 473us) server_negotiation.cc:299] Negotiation successful
0924 03:22:21.243930 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":349,"threads_started":1}
I20260924 03:22:21.247144 18631 raft_consensus.cc:348] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69ccdc0e31f41978c85f75a574a9cdc" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36271 } }
I20260924 03:22:21.247697 18631 raft_consensus.cc:374] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:21.247813 18631 raft_consensus.cc:729] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b69ccdc0e31f41978c85f75a574a9cdc, State: Initialized, Role: FOLLOWER
I20260924 03:22:21.248487 18631 consensus_queue.cc:260] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69ccdc0e31f41978c85f75a574a9cdc" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36271 } }
I20260924 03:22:21.248770 18631 raft_consensus.cc:388] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:22:21.248883 18631 raft_consensus.cc:482] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:22:21.249012 18631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:21.251364 18631 raft_consensus.cc:504] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69ccdc0e31f41978c85f75a574a9cdc" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36271 } }
I20260924 03:22:21.251943 18631 leader_election.cc:302] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b69ccdc0e31f41978c85f75a574a9cdc; no voters:
I20260924 03:22:21.252700 18631 leader_election.cc:288] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:22:21.252787 18636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:22:21.253458 18636 raft_consensus.cc:686] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [term 1 LEADER]: Becoming Leader. State: Replica: b69ccdc0e31f41978c85f75a574a9cdc, State: Running, Role: LEADER
I20260924 03:22:21.254072 18636 consensus_queue.cc:237] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69ccdc0e31f41978c85f75a574a9cdc" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36271 } }
I20260924 03:22:21.254776 18631 sys_catalog.cc:564] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:22:21.256801 18638 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b69ccdc0e31f41978c85f75a574a9cdc. Latest consensus state: current_term: 1 leader_uuid: "b69ccdc0e31f41978c85f75a574a9cdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69ccdc0e31f41978c85f75a574a9cdc" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36271 } } }
I20260924 03:22:21.257131 18638 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:21.257998 18642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:22:21.257849 18637 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b69ccdc0e31f41978c85f75a574a9cdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69ccdc0e31f41978c85f75a574a9cdc" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36271 } } }
I20260924 03:22:21.258244 18637 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:21.264112 18642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:22:21.274190 18642 catalog_manager.cc:1347] Generated new cluster ID: f9a116507c4e47289f3931db5e174a22
I20260924 03:22:21.274329 18642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:22:21.292132 18642 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:22:21.293618 18642 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:22:21.306897 18642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc: Generated new TSK 0
I20260924 03:22:21.307837 18642 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
I20260924 03:22:21.364499 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:36271
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:43807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:22:21.706550 18657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:21.706882 18657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:21.707010 18657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:21.714542 18657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:21.714701 18657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:21.714856 18657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:22:21.725258 18657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:43807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:36271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:21.726663 18657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:21.728016 18657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:21.741220 18662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:21.741178 18663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:21.741153 18665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:21.742492 18657 server_base.cc:1034] running on GCE node
I20260924 03:22:21.743232 18657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:21.744480 18657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:21.745746 18657 hybrid_clock.cc:648] HybridClock initialized: now 1790220141745713 us; error 89 us; skew 500 ppm
I20260924 03:22:21.746102 18657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:21 dist-test-slave-00qs krb5kdc[18551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220141, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:21.749496 18657 init.cc:377] Logged in from keytab as kudu/127.10.12.129@KRBTEST.COM (short username kudu)
I20260924 03:22:21.751067 18657 webserver.cc:460] Webserver started at http://127.10.12.129:39375/ using document root <none> and password file <none>
I20260924 03:22:21.751766 18657 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:21.752131 18657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:21.752467 18657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:21.755548 18657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "570afcb526ad4b689820d871fcbf066d"
format_stamp: "Formatted at 2026-09-24 03:22:21 on dist-test-slave-00qs"
server_key: "c4d41b7662f4697523edfbc0d85b5c54"
server_key_iv: "acd11cf66db3ac096edc6867c2e6cebe"
server_key_version: "encryptionkey@0"
I20260924 03:22:21.756556 18657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "570afcb526ad4b689820d871fcbf066d"
format_stamp: "Formatted at 2026-09-24 03:22:21 on dist-test-slave-00qs"
server_key: "c4d41b7662f4697523edfbc0d85b5c54"
server_key_iv: "acd11cf66db3ac096edc6867c2e6cebe"
server_key_version: "encryptionkey@0"
I20260924 03:22:21.762468 18657 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 03:22:21.766434 18672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:21.768105 18657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:22:21.768288 18657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "570afcb526ad4b689820d871fcbf066d"
format_stamp: "Formatted at 2026-09-24 03:22:21 on dist-test-slave-00qs"
server_key: "c4d41b7662f4697523edfbc0d85b5c54"
server_key_iv: "acd11cf66db3ac096edc6867c2e6cebe"
server_key_version: "encryptionkey@0"
I20260924 03:22:21.768517 18657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:21.784483 18657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:21.789012 18657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:21.789304 18657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:21.790372 18657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:22:21.791899 18657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:22:21.792032 18657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 03:22:21.792141 18657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:22:21.792186 18657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:21.835837 18657 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:34843
I20260924 03:22:21.835767 18784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:34843 every 8 connection(s)
I20260924 03:22:21.838300 18657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:22:21.839741 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 18657
I20260924 03:22:21.839941 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:22:21.840757 10290 external_mini_cluster.cc:1428] Setting key eefe315c48de435f09c7d1eaf271767e
WARNING: no policy specified for kudu/127.10.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.130@KRBTEST.COM" created.
Sep 24 03:22:21 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220141, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:22:21.881865 18788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:21.845495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:44003 (local address 127.10.12.190:36271)
0924 03:22:21.860445 (+ 14950us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:21.860450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:21.860478 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:22:21.860547 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:21.860551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:21.860631 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:21.860759 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:21.863532 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:21.864870 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:21.867807 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:21.868768 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:21.868975 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:21.875671 (+ 6696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:21.875678 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:21.875682 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:21.875724 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:21.878209 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:21.879316 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:21.879322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:21.879326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:21.879485 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:21.880000 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:21.880007 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:21.880010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:21.880308 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:21.880454 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:21.881247 (+ 793us) server_negotiation.cc:299] Negotiation successful
0924 03:22:21.881502 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14808,"thread_start_us":14682,"threads_started":1}
I20260924 03:22:21.882912 18787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:21.844906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.129:44003)
0924 03:22:21.845885 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:21.845942 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:21.848396 (+ 2454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:21.861271 (+ 12875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:21.861288 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:21.862210 (+ 922us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:21.863245 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:21.863278 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:21.865121 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:21.865128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:21.867395 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:21.867423 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:21.869095 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:21.869105 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:21.869393 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:21.870186 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:21.870211 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:21.875354 (+ 5143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:21.878415 (+ 3061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:21.878427 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:21.878449 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:21.879126 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:21.879612 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:21.879617 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:21.879619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:21.879831 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:21.880471 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:21.880480 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:21.880850 (+ 370us) client_negotiation.cc:769] Sending connection context
0924 03:22:21.881116 (+ 266us) client_negotiation.cc:240] Negotiation successful
0924 03:22:21.881692 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":361,"threads_started":1}
I20260924 03:22:21.884652 18785 heartbeater.cc:347] Connected to a master server at 127.10.12.190:36271
I20260924 03:22:21.885006 18785 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:21.885900 18785 heartbeater.cc:511] Master 127.10.12.190:36271 requested a full tablet report, sending...
I20260924 03:22:21.888815 18596 ts_manager.cc:194] Registered new tserver with Master: 570afcb526ad4b689820d871fcbf066d (127.10.12.129:34843)
I20260924 03:22:21.890643 18596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.129@KRBTEST.COM'} at 127.10.12.129:44003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
I20260924 03:22:21.907734 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:36271
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:43807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:22:22.239774 18791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:22.240136 18791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:22.240247 18791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:22.247736 18791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:22.247860 18791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:22.248008 18791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:22:22.258558 18791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:43807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:36271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:22.260262 18791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:22.261802 18791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:22.276350 18797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:22.276350 18796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:22.277136 18799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:22.277331 18791 server_base.cc:1034] running on GCE node
I20260924 03:22:22.278030 18791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:22.279110 18791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:22.280331 18791 hybrid_clock.cc:648] HybridClock initialized: now 1790220142280268 us; error 50 us; skew 500 ppm
I20260924 03:22:22.280717 18791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:22 dist-test-slave-00qs krb5kdc[18551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220142, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:22.284206 18791 init.cc:377] Logged in from keytab as kudu/127.10.12.130@KRBTEST.COM (short username kudu)
I20260924 03:22:22.285902 18791 webserver.cc:460] Webserver started at http://127.10.12.130:43785/ using document root <none> and password file <none>
I20260924 03:22:22.286767 18791 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:22.286912 18791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:22.287231 18791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:22.290584 18791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "51c3128c9e8f41ac8efd4b88ead87b1f"
format_stamp: "Formatted at 2026-09-24 03:22:22 on dist-test-slave-00qs"
server_key: "925fb832f4293a5883cf69a05b6d8895"
server_key_iv: "ad93916228d94b0e149cbc5f9cc8e037"
server_key_version: "encryptionkey@0"
I20260924 03:22:22.291457 18791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "51c3128c9e8f41ac8efd4b88ead87b1f"
format_stamp: "Formatted at 2026-09-24 03:22:22 on dist-test-slave-00qs"
server_key: "925fb832f4293a5883cf69a05b6d8895"
server_key_iv: "ad93916228d94b0e149cbc5f9cc8e037"
server_key_version: "encryptionkey@0"
I20260924 03:22:22.297299 18791 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:22:22.301496 18806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:22.303272 18791 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:22:22.303484 18791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "51c3128c9e8f41ac8efd4b88ead87b1f"
format_stamp: "Formatted at 2026-09-24 03:22:22 on dist-test-slave-00qs"
server_key: "925fb832f4293a5883cf69a05b6d8895"
server_key_iv: "ad93916228d94b0e149cbc5f9cc8e037"
server_key_version: "encryptionkey@0"
I20260924 03:22:22.303700 18791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:22.321441 18791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:22.325860 18791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:22.326159 18791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:22.327160 18791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:22:22.328630 18791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:22:22.328713 18791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:22.328837 18791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:22:22.328898 18791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:22.360538 18791 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:43769
I20260924 03:22:22.360580 18918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:43769 every 8 connection(s)
I20260924 03:22:22.361827 18791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:22:22.370296 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 18791
I20260924 03:22:22.370473 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:22:22.370939 10290 external_mini_cluster.cc:1428] Setting key b8759218de031072a9e5438a7147a2bf
Sep 24 03:22:22 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220142, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:22:22.385880 18788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:22.364591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:51541 (local address 127.10.12.190:36271)
0924 03:22:22.364790 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:22.364796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:22.366137 (+ 1341us) server_negotiation.cc:407] Connection header received
0924 03:22:22.367992 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:22.367998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:22.368097 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:22.368195 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:22.369921 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:22.370700 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:22.372876 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:22.373591 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:22.373732 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:22.377891 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:22.377897 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:22.377900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:22.377938 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:22.380623 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:22.381859 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:22.381867 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:22.381870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:22.382379 (+ 509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:22.383406 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:22.383411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:22.383414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:22.384123 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:22.384314 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:22.385228 (+ 914us) server_negotiation.cc:299] Negotiation successful
0924 03:22:22.385509 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:22:22.386920 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:22.365130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.130:51541)
0924 03:22:22.365942 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:22.365997 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:22.367642 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:22.368404 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:22.368412 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:22.369039 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:22.369651 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:22.369671 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:22.370887 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:22.370893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:22.372718 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:22.372737 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:22.373852 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:22.373860 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:22.374120 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:22.374893 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:22.375003 (+ 110us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:22.377595 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:22.380816 (+ 3221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:22.380825 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:22.380852 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:22.381556 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:22.382789 (+ 1233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:22.382796 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:22.382799 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:22.383226 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:22.384347 (+ 1121us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:22.384359 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:22.384792 (+ 433us) client_negotiation.cc:769] Sending connection context
0924 03:22:22.385081 (+ 289us) client_negotiation.cc:240] Negotiation successful
0924 03:22:22.385684 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":298,"threads_started":1}
I20260924 03:22:22.388687 18919 heartbeater.cc:347] Connected to a master server at 127.10.12.190:36271
I20260924 03:22:22.389065 18919 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:22.390087 18919 heartbeater.cc:511] Master 127.10.12.190:36271 requested a full tablet report, sending...
I20260924 03:22:22.392046 18596 ts_manager.cc:194] Registered new tserver with Master: 51c3128c9e8f41ac8efd4b88ead87b1f (127.10.12.130:43769)
WARNING: no policy specified for kudu/127.10.12.131@KRBTEST.COM; defaulting to no policy
I20260924 03:22:22.393148 18596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.130@KRBTEST.COM'} at 127.10.12.130:51541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
I20260924 03:22:22.424597 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:36271
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:43807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:22:22.768122 18924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:22.768417 18924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:22.768527 18924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:22.776453 18924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:22.776575 18924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:22.776702 18924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:22:22.786693 18924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:43807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:36271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:22.787981 18924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:22.789285 18924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:22.802985 18930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:22.803020 18932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:22.803061 18929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:22.803700 18924 server_base.cc:1034] running on GCE node
I20260924 03:22:22.804471 18924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:22.805589 18924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:22.806834 18924 hybrid_clock.cc:648] HybridClock initialized: now 1790220142806755 us; error 67 us; skew 500 ppm
I20260924 03:22:22.807183 18924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:22 dist-test-slave-00qs krb5kdc[18551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220142, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:22.810585 18924 init.cc:377] Logged in from keytab as kudu/127.10.12.131@KRBTEST.COM (short username kudu)
I20260924 03:22:22.812213 18924 webserver.cc:460] Webserver started at http://127.10.12.131:45157/ using document root <none> and password file <none>
I20260924 03:22:22.812920 18924 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:22.813041 18924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:22.813350 18924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:22.816480 18924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "0a96c7218048420085bff32e24ea448f"
format_stamp: "Formatted at 2026-09-24 03:22:22 on dist-test-slave-00qs"
server_key: "437877ad15a68f489bb6e617944dab1a"
server_key_iv: "15d99bfed41a1ecc496c7477c30a0926"
server_key_version: "encryptionkey@0"
I20260924 03:22:22.817348 18924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "0a96c7218048420085bff32e24ea448f"
format_stamp: "Formatted at 2026-09-24 03:22:22 on dist-test-slave-00qs"
server_key: "437877ad15a68f489bb6e617944dab1a"
server_key_iv: "15d99bfed41a1ecc496c7477c30a0926"
server_key_version: "encryptionkey@0"
I20260924 03:22:22.824085 18924 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260924 03:22:22.828166 18939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:22.830155 18924 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:22:22.830344 18924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "0a96c7218048420085bff32e24ea448f"
format_stamp: "Formatted at 2026-09-24 03:22:22 on dist-test-slave-00qs"
server_key: "437877ad15a68f489bb6e617944dab1a"
server_key_iv: "15d99bfed41a1ecc496c7477c30a0926"
server_key_version: "encryptionkey@0"
I20260924 03:22:22.830580 18924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:22.846604 18924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:22.851395 18924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:22.851699 18924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:22.852758 18924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:22:22.854231 18924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:22:22.854317 18924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:22.854461 18924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:22:22.854532 18924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:22.889669 18924 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:36177
I20260924 03:22:22.889710 19051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:36177 every 8 connection(s)
I20260924 03:22:22.891256 18924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:22:22.894670 18785 heartbeater.cc:503] Master 127.10.12.190:36271 was elected leader, sending a full tablet report...
I20260924 03:22:22.898041 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 18924
I20260924 03:22:22.898206 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:22:22.898749 10290 external_mini_cluster.cc:1428] Setting key 69525d873f8ca562b19ccc3dbe678130
Sep 24 03:22:22 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220142, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:22:22.916584 19054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:22.894558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:41361 (local address 127.10.12.190:36271)
0924 03:22:22.894975 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:22.894982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:22.896191 (+ 1209us) server_negotiation.cc:407] Connection header received
0924 03:22:22.898142 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:22.898148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:22.898244 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:22.898347 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:22.900805 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:22.903179 (+ 2374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:22.905267 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:22.905847 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:22.906018 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:22.910143 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:22.910149 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:22.910152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:22.910197 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:22.912853 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:22.913751 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:22.913755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:22.913757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:22.913893 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:22.914390 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:22.914394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:22.914397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:22.914765 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:22.914920 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:22.916027 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0924 03:22:22.916270 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":215,"threads_started":1}
I20260924 03:22:22.917544 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:22.895245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.131:41361)
0924 03:22:22.896038 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:22.896088 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:22.897834 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:22.898670 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:22.898683 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:22.899654 (+ 971us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:22.900499 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:22.900615 (+ 116us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:22.903475 (+ 2860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:22.903482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:22.904967 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:22.904986 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:22.906164 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:22.906170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:22.906430 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:22.907218 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:22.907252 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:22.909758 (+ 2506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:22.913038 (+ 3280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:22.913047 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:22.913064 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:22.913579 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:22.914010 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:22.914015 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:22.914018 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:22.914235 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:22.914945 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:22.914954 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:22.915444 (+ 490us) client_negotiation.cc:769] Sending connection context
0924 03:22:22.915796 (+ 352us) client_negotiation.cc:240] Negotiation successful
0924 03:22:22.916375 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":293,"threads_started":1}
I20260924 03:22:22.919059 19052 heartbeater.cc:347] Connected to a master server at 127.10.12.190:36271
I20260924 03:22:22.919345 19052 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:22.920137 19052 heartbeater.cc:511] Master 127.10.12.190:36271 requested a full tablet report, sending...
I20260924 03:22:22.921660 18596 ts_manager.cc:194] Registered new tserver with Master: 0a96c7218048420085bff32e24ea448f (127.10.12.131:36177)
I20260924 03:22:22.922528 18596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.131@KRBTEST.COM'} at 127.10.12.131:41361
I20260924 03:22:22.928747 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.129.keytab.
Entry for principal oryx/127.10.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.129.keytab.
WARNING: no policy specified for oryx/127.10.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.130.keytab.
Entry for principal oryx/127.10.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.130.keytab.
WARNING: no policy specified for oryx/127.10.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.131.keytab.
Entry for principal oryx/127.10.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.131.keytab.
I20260924 03:22:23.076802 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 18657
Sep 24 03:22:23 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.129@KRBTEST.COM
I20260924 03:22:23.094154 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:23.078016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56282 (local address 127.10.12.129:34843)
0924 03:22:23.078636 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:23.078644 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:23.078673 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:22:23.078737 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:23.078744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:23.078955 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:23.079106 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:23.080504 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:23.081584 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:23.082865 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:23.083840 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:23.084138 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:23.088339 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:23.088354 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:23.088371 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:23.088414 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:23.091192 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:23.092091 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:23.092099 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:23.092101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:23.092246 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:23.092671 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:23.092675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:23.092678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:23.093090 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:23.093375 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:23.093634 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 03:22:23.093859 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":301,"threads_started":1}
I20260924 03:22:23.096179 18780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:23.180030 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 18657
I20260924 03:22:23.193435 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 18791
Sep 24 03:22:23 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.130@KRBTEST.COM
I20260924 03:22:23.210937 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:23.194484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38406 (local address 127.10.12.130:43769)
0924 03:22:23.194973 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:23.194980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:23.195006 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:22:23.195078 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:23.195084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:23.195306 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:23.195456 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:23.196818 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:23.197848 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:23.199185 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:23.199778 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:23.199910 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:23.204777 (+ 4867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:23.204794 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:23.204811 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:23.204853 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:23.207865 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:23.208749 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:23.208758 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:23.208760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:23.208913 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:23.209288 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:23.209291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:23.209293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:23.209697 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:23.210037 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:23.210297 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 03:22:23.210531 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":233,"threads_started":1}
I20260924 03:22:23.212399 18914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:23.296224 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 18791
I20260924 03:22:23.310280 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 18924
Sep 24 03:22:23 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.131@KRBTEST.COM
I20260924 03:22:23.328822 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:23.311489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39892 (local address 127.10.12.131:36177)
0924 03:22:23.312274 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:23.312282 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:23.312312 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:22:23.312402 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:23.312409 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:23.312639 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:23.312780 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:23.314274 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:23.315589 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:23.317088 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:23.317771 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:23.317904 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:23.322542 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:23.322556 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:23.322573 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:23.322616 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:23.325730 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:23.326576 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:23.326583 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:23.326586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:23.326725 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:23.327072 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:23.327076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:23.327078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:23.327475 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:23.327717 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:23.328196 (+ 479us) server_negotiation.cc:299] Negotiation successful
0924 03:22:23.328519 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":537,"threads_started":1}
I20260924 03:22:23.330315 19047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:23.414058 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 18924
I20260924 03:22:23.427069 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 18564
I20260924 03:22:23.428233 18626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:23.530790 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 18564
I20260924 03:22:23.547719 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:36271
--webserver_interface=127.10.12.190
--webserver_port=46455
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:43807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:36271
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:22:23.881732 19070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:23.882050 19070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:23.882139 19070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:23.889602 19070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:22:23.889727 19070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:23.889776 19070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:23.889828 19070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:22:23.889873 19070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:22:23.901418 19070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:43807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:36271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:36271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=46455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:23.902807 19070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:23.904278 19070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:23.916875 19078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:23.917220 19075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:23.917029 19076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:23.918885 19070 server_base.cc:1034] running on GCE node
I20260924 03:22:23.919744 19070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:23.921376 19070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:23.922611 19070 hybrid_clock.cc:648] HybridClock initialized: now 1790220143922554 us; error 59 us; skew 500 ppm
I20260924 03:22:23.922967 19070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:23 dist-test-slave-00qs krb5kdc[18551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220143, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:23.926146 19070 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:22:23.927766 19070 webserver.cc:460] Webserver started at http://127.10.12.190:46455/ using document root <none> and password file <none>
I20260924 03:22:23.928547 19070 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:23.928668 19070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:23.935024 19070 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 03:22:23.939111 19085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:23.940889 19070 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260924 03:22:23.941136 19070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "b69ccdc0e31f41978c85f75a574a9cdc"
format_stamp: "Formatted at 2026-09-24 03:22:21 on dist-test-slave-00qs"
server_key: "cc4db452e170dc98d79bb83160f84b66"
server_key_iv: "72dfd60238890f27ad33f39bda0af738"
server_key_version: "encryptionkey@0"
I20260924 03:22:23.942160 19070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:23.955924 19070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:23.960703 19070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:23.960985 19070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:23.985282 19070 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:36271
I20260924 03:22:23.985301 19136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:36271 every 8 connection(s)
I20260924 03:22:23.986886 19070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:22:23.991030 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 19070
I20260924 03:22:23.991860 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:34843
--local_ip_for_outbound_sockets=127.10.12.129
--tserver_master_addrs=127.10.12.190:36271
--webserver_port=39375
--webserver_interface=127.10.12.129
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:43807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.12.129
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:22:23.994491 19137 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:22:23.998347 19137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc: Bootstrap starting.
I20260924 03:22:24.017947 19137 log.cc:824] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc: Log is configured to *not* fsync() on all Append() calls
I20260924 03:22:24.027544 19137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:22:24.028132 19137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc: Bootstrap complete.
I20260924 03:22:24.037298 19137 raft_consensus.cc:348] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69ccdc0e31f41978c85f75a574a9cdc" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36271 } }
I20260924 03:22:24.038210 19137 raft_consensus.cc:729] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b69ccdc0e31f41978c85f75a574a9cdc, State: Initialized, Role: FOLLOWER
I20260924 03:22:24.038823 19137 consensus_queue.cc:260] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69ccdc0e31f41978c85f75a574a9cdc" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36271 } }
I20260924 03:22:24.039065 19137 raft_consensus.cc:388] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:22:24.039175 19137 raft_consensus.cc:482] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:22:24.039328 19137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:22:24.043311 19137 raft_consensus.cc:504] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69ccdc0e31f41978c85f75a574a9cdc" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36271 } }
I20260924 03:22:24.043856 19137 leader_election.cc:302] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b69ccdc0e31f41978c85f75a574a9cdc; no voters:
I20260924 03:22:24.044525 19137 leader_election.cc:288] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 03:22:24.044679 19141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [term 2 FOLLOWER]: Leader election won for term 2
I20260924 03:22:24.045816 19141 raft_consensus.cc:686] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [term 2 LEADER]: Becoming Leader. State: Replica: b69ccdc0e31f41978c85f75a574a9cdc, State: Running, Role: LEADER
I20260924 03:22:24.046535 19141 consensus_queue.cc:237] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69ccdc0e31f41978c85f75a574a9cdc" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36271 } }
I20260924 03:22:24.046923 19137 sys_catalog.cc:564] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:22:24.052721 19142 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b69ccdc0e31f41978c85f75a574a9cdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69ccdc0e31f41978c85f75a574a9cdc" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36271 } } }
I20260924 03:22:24.053397 19142 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:24.053058 19143 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b69ccdc0e31f41978c85f75a574a9cdc. Latest consensus state: current_term: 2 leader_uuid: "b69ccdc0e31f41978c85f75a574a9cdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69ccdc0e31f41978c85f75a574a9cdc" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36271 } } }
I20260924 03:22:24.054108 19143 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:24.058692 19149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:22:24.065320 19149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:22:24.067114 19149 catalog_manager.cc:1259] Loaded cluster ID: f9a116507c4e47289f3931db5e174a22
I20260924 03:22:24.067368 19149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:22:24.072198 19149 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:22:24.074251 19149 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b69ccdc0e31f41978c85f75a574a9cdc: Loaded TSK: 0
I20260924 03:22:24.076654 19149 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 03:22:24.372226 19139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:24.372550 19139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:24.372669 19139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:24.380685 19139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:24.380823 19139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:24.380975 19139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:22:24.391768 19139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:43807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.129.keytab
--principal=oryx/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:34843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=39375
--tserver_master_addrs=127.10.12.190:36271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:24.393265 19139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:24.394682 19139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:24.408672 19164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:24.408646 19163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:24.408870 19166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:24.410580 19139 server_base.cc:1034] running on GCE node
I20260924 03:22:24.411293 19139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:24.412406 19139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:24.413663 19139 hybrid_clock.cc:648] HybridClock initialized: now 1790220144413597 us; error 59 us; skew 500 ppm
I20260924 03:22:24.414026 19139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:24 dist-test-slave-00qs krb5kdc[18551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220144, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:24.417305 19139 init.cc:377] Logged in from keytab as oryx/127.10.12.129@KRBTEST.COM (short username oryx)
I20260924 03:22:24.419210 19139 webserver.cc:460] Webserver started at http://127.10.12.129:39375/ using document root <none> and password file <none>
I20260924 03:22:24.420034 19139 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:24.420163 19139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:24.426723 19139 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:22:24.431344 19173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:24.433755 19139 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260924 03:22:24.434134 19139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "570afcb526ad4b689820d871fcbf066d"
format_stamp: "Formatted at 2026-09-24 03:22:21 on dist-test-slave-00qs"
server_key: "c4d41b7662f4697523edfbc0d85b5c54"
server_key_iv: "acd11cf66db3ac096edc6867c2e6cebe"
server_key_version: "encryptionkey@0"
I20260924 03:22:24.435665 19139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:24.463006 19139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:24.468061 19139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:24.468370 19139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:24.469628 19139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:22:24.471266 19139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:22:24.471400 19139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:24.471495 19139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:22:24.471571 19139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:24.504189 19139 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:34843
I20260924 03:22:24.504235 19285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:34843 every 8 connection(s)
I20260924 03:22:24.505731 19139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:22:24.516203 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 19139
I20260924 03:22:24.516979 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:43769
--local_ip_for_outbound_sockets=127.10.12.130
--tserver_master_addrs=127.10.12.190:36271
--webserver_port=43785
--webserver_interface=127.10.12.130
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:43807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.12.130
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:22:24 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:24 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 03:22:24.860571 19290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:24.860874 19290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:24.860996 19290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:24.868533 19290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:24.868664 19290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:24.868808 19290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:22:24.879256 19290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:43807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.130.keytab
--principal=oryx/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:43769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=43785
--tserver_master_addrs=127.10.12.190:36271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:24.880710 19290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:24.882033 19290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:24.895980 19296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:24.896057 19297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:24.896229 19299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:24.906193 19290 server_base.cc:1034] running on GCE node
I20260924 03:22:24.907258 19290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:24.908841 19290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:24.910145 19290 hybrid_clock.cc:648] HybridClock initialized: now 1790220144910092 us; error 42 us; skew 500 ppm
I20260924 03:22:24.910625 19290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:24 dist-test-slave-00qs krb5kdc[18551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220144, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:24.916217 19290 init.cc:377] Logged in from keytab as oryx/127.10.12.130@KRBTEST.COM (short username oryx)
I20260924 03:22:24.918709 19290 webserver.cc:460] Webserver started at http://127.10.12.130:43785/ using document root <none> and password file <none>
I20260924 03:22:24.920117 19290 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:24.920450 19290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:24.929589 19290 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 03:22:24.934574 19306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:24.937120 19290 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.004s
I20260924 03:22:24.937397 19290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "51c3128c9e8f41ac8efd4b88ead87b1f"
format_stamp: "Formatted at 2026-09-24 03:22:22 on dist-test-slave-00qs"
server_key: "925fb832f4293a5883cf69a05b6d8895"
server_key_iv: "ad93916228d94b0e149cbc5f9cc8e037"
server_key_version: "encryptionkey@0"
I20260924 03:22:24.939121 19290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:24.968379 19290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:24.975390 19290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:24.975769 19290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:24.977396 19290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:22:24.979593 19290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:22:24.979740 19290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:24.979874 19290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:22:24.980000 19290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:25.021399 19290 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:43769
I20260924 03:22:25.021416 19418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:43769 every 8 connection(s)
I20260924 03:22:25.023703 19290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:22:25.025291 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 19290
I20260924 03:22:25.026003 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:36177
--local_ip_for_outbound_sockets=127.10.12.131
--tserver_master_addrs=127.10.12.190:36271
--webserver_port=45157
--webserver_interface=127.10.12.131
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:43807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.12.131
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:22:25 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:25 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 03:22:25.375663 19423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:25.376099 19423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:25.376240 19423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:25.383731 19423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:25.383879 19423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:25.384071 19423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:22:25.394694 19423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:43807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/krb5kdc/oryx_127.10.12.131.keytab
--principal=oryx/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:36177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=45157
--tserver_master_addrs=127.10.12.190:36271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:25.396217 19423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:25.397554 19423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:25.414425 19430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:25.416471 19429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:25.421748 19432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:25.422618 19423 server_base.cc:1034] running on GCE node
I20260924 03:22:25.423632 19423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:25.425313 19423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:25.426702 19423 hybrid_clock.cc:648] HybridClock initialized: now 1790220145426631 us; error 65 us; skew 500 ppm
I20260924 03:22:25.427175 19423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:25 dist-test-slave-00qs krb5kdc[18551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220145, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:25.432551 19423 init.cc:377] Logged in from keytab as oryx/127.10.12.131@KRBTEST.COM (short username oryx)
I20260924 03:22:25.438804 19423 webserver.cc:460] Webserver started at http://127.10.12.131:45157/ using document root <none> and password file <none>
I20260924 03:22:25.440152 19423 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:25.440371 19423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:25.450428 19423 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 03:22:25.456141 19439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:25.459159 19423 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260924 03:22:25.459568 19423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "0a96c7218048420085bff32e24ea448f"
format_stamp: "Formatted at 2026-09-24 03:22:22 on dist-test-slave-00qs"
server_key: "437877ad15a68f489bb6e617944dab1a"
server_key_iv: "15d99bfed41a1ecc496c7477c30a0926"
server_key_version: "encryptionkey@0"
I20260924 03:22:25.461722 19423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:25.530493 19423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:25.537249 19423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:25.537627 19423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:25.539145 19423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:22:25.541622 19423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:22:25.541777 19423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:25.541913 19423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:22:25.542001 19423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:25.591284 19423 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:36177
I20260924 03:22:25.591321 19551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:36177 every 8 connection(s)
I20260924 03:22:25.593663 19423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:22:25.602262 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 19423
Sep 24 03:22:25 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:25 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:25.646733 19555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:25.603407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36210 (local address 127.10.12.190:36271)
0924 03:22:25.620956 (+ 17549us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:25.620962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:25.620994 (+ 32us) server_negotiation.cc:407] Connection header received
0924 03:22:25.621079 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:25.621085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:25.621183 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:25.623845 (+ 2662us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:25.625549 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:25.626795 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:25.629424 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:25.630098 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:25.630235 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:25.634071 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:25.634082 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:25.634107 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:25.634167 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:25.638661 (+ 4494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:25.639784 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:25.639795 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:25.639806 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:25.640167 (+ 361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:25.640663 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:25.640667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:25.640671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:25.641878 (+ 1207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:25.643891 (+ 2013us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:25.644218 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 03:22:25.645275 (+ 1057us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17435,"thread_start_us":16854,"threads_started":1}
W20260924 03:22:25.687680 19289 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.167s user 0.000s sys 0.016s
W20260924 03:22:25.689492 19288 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:24.508812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:56903 (local address 127.10.12.190:36271)
0924 03:22:24.509579 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:24.509595 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:24.509984 (+ 389us) server_negotiation.cc:407] Connection header received
0924 03:22:24.512313 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:24.512329 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:24.512711 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:24.513018 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:24.514634 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:24.515883 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:24.518279 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:24.519341 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:24.519543 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:25.689146 (+1169603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.129:56903: BlockingRecv error: recv got EOF from 127.10.12.129:56903 (error 108)
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":262,"threads_started":1}
I20260924 03:22:25.690151 19289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:24.508860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.129:56903)
0924 03:22:24.509809 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:24.509854 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:24.511616 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:24.513214 (+ 1598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:24.513223 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:24.513827 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:24.514414 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:24.514434 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:24.516098 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:24.516105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:24.518072 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:24.518090 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:24.519909 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:24.519918 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:24.520250 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:24.521061 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:24.521094 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:25.688841 (+1167747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":519,"threads_started":1}
W20260924 03:22:25.691380 19286 heartbeater.cc:644] Failed to heartbeat to 127.10.12.190:36271 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.12.190:36271: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Sep 24 03:22:25 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:25 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:25.707123 19289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:25.692314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.129:45141)
0924 03:22:25.693059 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:25.693084 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:25.693250 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:25.693763 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:25.693769 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:25.694353 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:25.694612 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:25.694626 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:25.696373 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:25.696380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:25.698180 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:25.698202 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:25.699424 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:25.699432 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:25.699655 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:25.700561 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:25.700593 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:25.706726 (+ 6133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":576}
W20260924 03:22:25.707373 19288 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:25.692362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:45141 (local address 127.10.12.190:36271)
0924 03:22:25.692580 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:25.692587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:25.693177 (+ 590us) server_negotiation.cc:407] Connection header received
0924 03:22:25.693394 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:25.693400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:25.693550 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:25.693706 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:25.694803 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:25.696187 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:25.698398 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:25.699152 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:25.699347 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:25.707073 (+ 7726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.129:45141: BlockingRecv error: recv got EOF from 127.10.12.129:45141 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 24 03:22:25 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:25 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:25.727310 19289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:25.708454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.129:36087)
0924 03:22:25.712382 (+ 3928us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:25.712415 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:25.712733 (+ 318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:25.713397 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:25.713402 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:25.714767 (+ 1365us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:25.715069 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:25.715086 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:25.716924 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:25.716931 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:25.718672 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:25.718693 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:25.720197 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:25.720204 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:25.720451 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:25.721315 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:25.721345 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:25.726922 (+ 5577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3769,"thread_start_us":16957,"threads_started":1}
W20260924 03:22:25.727629 19288 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:25.708551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:36087 (local address 127.10.12.190:36271)
0924 03:22:25.708926 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:25.708951 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:25.712574 (+ 3623us) server_negotiation.cc:407] Connection header received
0924 03:22:25.712909 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:25.712916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:25.713040 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:25.713125 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:25.715293 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:25.716653 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:25.718923 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:25.719710 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:25.719828 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:25.727297 (+ 7469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.129:36087: BlockingRecv error: recv got EOF from 127.10.12.129:36087 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 03:22:25.727730 19286 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:22:26.202726 19422 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.153s user 0.003s sys 0.014s
W20260924 03:22:26.204528 19421 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:25.028084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:37875 (local address 127.10.12.190:36271)
0924 03:22:25.040108 (+ 12024us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:25.040116 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:25.040147 (+ 31us) server_negotiation.cc:407] Connection header received
0924 03:22:25.040236 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:25.040242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:25.040363 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:25.040516 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:25.043065 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:25.044496 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:25.047361 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:25.048270 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:25.048435 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:26.204208 (+1155773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.130:37875: BlockingRecv error: recv got EOF from 127.10.12.130:37875 (error 108)
Metrics: {"server-negotiator.queue_time_us":11800,"thread_start_us":11611,"threads_started":1}
I20260924 03:22:26.205469 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:25.028984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.130:37875)
0924 03:22:25.035709 (+ 6725us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:25.035799 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:25.038789 (+ 2990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:25.041024 (+ 2235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:25.041038 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:25.041878 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:25.042780 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:25.042815 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:25.044764 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:25.044770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:25.047163 (+ 2393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:25.047186 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:25.048557 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:25.048565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:25.048884 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:25.049780 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:25.049817 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:26.203935 (+1154118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":6220,"thread_start_us":5904,"threads_started":1}
W20260924 03:22:26.206857 19419 heartbeater.cc:644] Failed to heartbeat to 127.10.12.190:36271 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.12.190:36271: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Sep 24 03:22:26 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:26 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:26.222100 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:26.208359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.130:51401)
0924 03:22:26.208596 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:26.208615 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:26.208746 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:26.209060 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:26.209065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:26.209450 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:26.209621 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:26.209631 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:26.210647 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:26.210653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:26.211624 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:26.211638 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:26.212449 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:26.212455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:26.212633 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:26.217409 (+ 4776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:26.217437 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:26.221780 (+ 4343us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84}
W20260924 03:22:26.222213 19421 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:26.208386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:51401 (local address 127.10.12.190:36271)
0924 03:22:26.208573 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:26.208580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:26.208667 (+ 87us) server_negotiation.cc:407] Connection header received
0924 03:22:26.208858 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:26.208863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:26.208932 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:26.209015 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:26.209769 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:26.210487 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:26.211777 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:26.212259 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:26.212390 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:26.222008 (+ 9618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.130:51401: BlockingRecv error: recv got EOF from 127.10.12.130:51401 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 03:22:26 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:26 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:26.234866 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:26.223120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.130:40635)
0924 03:22:26.223412 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:26.223433 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:26.223575 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:26.223970 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:26.223976 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:26.224486 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:26.224739 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:26.224754 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:26.225958 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:26.225964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:26.227747 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:26.227770 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:26.228664 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:26.228671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:26.228916 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:26.229625 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:26.229652 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:26.234513 (+ 4861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164}
W20260924 03:22:26.234906 19421 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:26.223248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:40635 (local address 127.10.12.190:36271)
0924 03:22:26.223388 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:26.223394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:26.223486 (+ 92us) server_negotiation.cc:407] Connection header received
0924 03:22:26.223699 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:26.223702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:26.223762 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:26.223842 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:26.224939 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:26.225803 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:26.227974 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:26.228487 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:26.228597 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:26.234712 (+ 6115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.130:40635: BlockingRecv error: recv got EOF from 127.10.12.130:40635 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 03:22:26.235347 19419 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:22:26.672304 19554 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.031s user 0.007s sys 0.012s
W20260924 03:22:26.673491 19556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:25.598300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:60425 (local address 127.10.12.190:36271)
0924 03:22:25.620589 (+ 22289us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:25.620598 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:25.625298 (+ 4700us) server_negotiation.cc:407] Connection header received
0924 03:22:25.629344 (+ 4046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:25.629351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:25.629457 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:25.629578 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:25.633770 (+ 4192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:25.635390 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:25.638695 (+ 3305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:25.639464 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:25.639592 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:26.673264 (+1033672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.131:60425: BlockingRecv error: recv got EOF from 127.10.12.131:60425 (error 108)
Metrics: {"server-negotiator.queue_time_us":22126,"thread_start_us":22438,"threads_started":1}
I20260924 03:22:26.673925 19554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:25.597646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.131:60425)
0924 03:22:25.624838 (+ 27192us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:25.624919 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:25.628956 (+ 4037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:25.631150 (+ 2194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:25.631170 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:25.632292 (+ 1122us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:25.633419 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:25.633456 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:25.635724 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:25.635731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:25.638383 (+ 2652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:25.638415 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:25.640157 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:25.640165 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:25.640550 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:25.641430 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:25.641475 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:26.673030 (+1031555us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26719,"thread_start_us":26414,"threads_started":1}
W20260924 03:22:26.674827 19552 heartbeater.cc:644] Failed to heartbeat to 127.10.12.190:36271 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.12.190:36271: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Sep 24 03:22:26 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:26 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:26.685279 19554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:26.675477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.131:45419)
0924 03:22:26.675732 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:26.675749 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:26.675852 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:26.676404 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:26.676408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:26.676745 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:26.676929 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:26.676938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:26.678400 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:26.678405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:26.679455 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:26.679472 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:26.680333 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:26.680339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:26.680484 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:26.681095 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:26.681115 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:26.685046 (+ 3931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138}
W20260924 03:22:26.685549 19556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:26.675568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:45419 (local address 127.10.12.190:36271)
0924 03:22:26.675771 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:26.675776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:26.675794 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:22:26.676135 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:26.676139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:26.676202 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:26.676285 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:26.677175 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:26.678201 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:26.679627 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:26.680128 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:26.680251 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:26.685315 (+ 5064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.131:45419: BlockingRecv error: recv got EOF from 127.10.12.131:45419 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 24 03:22:26 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:26 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:26.694765 19554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:26.685943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.131:59155)
0924 03:22:26.686109 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:26.686129 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:26.686223 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:26.686497 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:26.686500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:26.686815 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:26.687051 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:26.687062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:26.688140 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:26.688146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:26.689143 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:26.689158 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:26.689878 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:26.689884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:26.690094 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:26.690684 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:26.690704 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:26.694458 (+ 3754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 03:22:26.695081 19552 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:22:26.695379 19556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:26.686008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:59155 (local address 127.10.12.190:36271)
0924 03:22:26.686214 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:26.686219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:26.686235 (+ 16us) server_negotiation.cc:407] Connection header received
0924 03:22:26.686305 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:26.686308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:26.686377 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:26.686455 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:26.687198 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:26.687906 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:26.689300 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:26.689717 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:26.689826 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:26.694778 (+ 4952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.131:59155: BlockingRecv error: recv got EOF from 127.10.12.131:59155 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Sep 24 03:22:26 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:26 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:26.738152 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:26.728425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.129:46955)
0924 03:22:26.728955 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:26.728972 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:26.729115 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:26.729449 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:26.729454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:26.729985 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:26.730173 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:26.730186 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:26.731388 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:26.731392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:26.732540 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:26.732557 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:26.733472 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:26.733479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:26.733636 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:26.734268 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:26.734287 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:26.737894 (+ 3607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":293,"threads_started":1}
W20260924 03:22:26.738430 19556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:26.728576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:46955 (local address 127.10.12.190:36271)
0924 03:22:26.728809 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:26.728815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:26.729063 (+ 248us) server_negotiation.cc:407] Connection header received
0924 03:22:26.729237 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:26.729241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:26.729313 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:26.729397 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:26.730350 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:26.731225 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:26.732714 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:26.733277 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:26.733384 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:26.738208 (+ 4824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.129:46955: BlockingRecv error: recv got EOF from 127.10.12.129:46955 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 24 03:22:27 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:27 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:27.246178 19561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:27.236274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.130:55425)
0924 03:22:27.236817 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:27.236833 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:27.236962 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:27.237258 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:27.237261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:27.237732 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:27.237879 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:27.237888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:27.239544 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:27.239550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:27.240795 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:27.240812 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:27.241624 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:27.241630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:27.241760 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:27.242325 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:27.242345 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:27.245884 (+ 3539us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":305,"threads_started":1}
W20260924 03:22:27.246383 19556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:27.236414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:55425 (local address 127.10.12.190:36271)
0924 03:22:27.236621 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:27.236626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:27.236950 (+ 324us) server_negotiation.cc:407] Connection header received
0924 03:22:27.237067 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:27.237071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:27.237135 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:27.237229 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:27.238138 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:27.239347 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:27.240937 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:27.241416 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:27.241517 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:27.246167 (+ 4650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.130:55425: BlockingRecv error: recv got EOF from 127.10.12.130:55425 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 24 03:22:27 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:27 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:27.705808 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:27.695884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.131:60827)
0924 03:22:27.696642 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:27.696660 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:27.696814 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:27.697163 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:27.697168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:27.697736 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:27.697885 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:27.697895 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:27.698990 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:27.698994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:27.700115 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:27.700130 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:27.700942 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:27.700949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:27.701116 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:27.701712 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:27.701857 (+ 145us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:27.705568 (+ 3711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":392,"threads_started":1}
W20260924 03:22:27.706108 19556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:27.696213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:60827 (local address 127.10.12.190:36271)
0924 03:22:27.696437 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:27.696444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:27.696772 (+ 328us) server_negotiation.cc:407] Connection header received
0924 03:22:27.696933 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:27.696937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:27.697027 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:27.697111 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:27.698044 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:27.698849 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:27.700249 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:27.700763 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:27.700876 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:27.705840 (+ 4964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.131:60827: BlockingRecv error: recv got EOF from 127.10.12.131:60827 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 24 03:22:27 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:27 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:27.749720 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:27.739205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.129:57437)
0924 03:22:27.739870 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:27.739996 (+ 126us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:27.740158 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:27.740489 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:27.740493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:27.741015 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:27.741231 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:27.741241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:27.742263 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:27.742267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:27.743203 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:27.743215 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:27.743928 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:27.743934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:27.744133 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:27.744739 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:27.744758 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:27.749386 (+ 4628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":352,"threads_started":1}
W20260924 03:22:27.749907 19556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:27.739319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:57437 (local address 127.10.12.190:36271)
0924 03:22:27.739574 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:27.739581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:27.740082 (+ 501us) server_negotiation.cc:407] Connection header received
0924 03:22:27.740278 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:27.740283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:27.740343 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:27.740461 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:27.741380 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:27.742126 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:27.743356 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:27.743760 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:27.743870 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:27.749659 (+ 5789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.129:57437: BlockingRecv error: recv got EOF from 127.10.12.129:57437 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Sep 24 03:22:28 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:28 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:28.256392 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:28.247231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.130:40169)
0924 03:22:28.247732 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:28.247751 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:28.247864 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:28.248249 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:28.248252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:28.248657 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:28.248829 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:28.248838 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:28.249831 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:28.249835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:28.250782 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:28.250795 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:28.251490 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:28.251495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:28.251621 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:28.252312 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:28.252330 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:28.256138 (+ 3808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":299,"threads_started":1}
W20260924 03:22:28.256697 19556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:28.247257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:40169 (local address 127.10.12.190:36271)
0924 03:22:28.247451 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:28.247456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:28.247815 (+ 359us) server_negotiation.cc:407] Connection header received
0924 03:22:28.248059 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:28.248063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:28.248126 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:28.248209 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:28.248960 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:28.249696 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:28.250917 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:28.251332 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:28.251428 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:28.256444 (+ 5016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.130:40169: BlockingRecv error: recv got EOF from 127.10.12.130:40169 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 03:22:28 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:28 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:28.716691 19565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:28.706832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.131:33643)
0924 03:22:28.707304 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:28.707319 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:28.707458 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:28.707868 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:28.707873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:28.708362 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:28.708545 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:28.708555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:28.709833 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:28.709840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:28.711156 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:28.711176 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:28.711910 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:28.711914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:28.712076 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:28.712667 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:28.712683 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:28.716412 (+ 3729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":267,"threads_started":1}
W20260924 03:22:28.717072 19556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:28.706974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:33643 (local address 127.10.12.190:36271)
0924 03:22:28.707178 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:28.707183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:28.707414 (+ 231us) server_negotiation.cc:407] Connection header received
0924 03:22:28.707639 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:28.707643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:28.707712 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:28.707803 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:28.708703 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:28.709523 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:28.711336 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:28.711763 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:28.711867 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:28.716787 (+ 4920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.131:33643: BlockingRecv error: recv got EOF from 127.10.12.131:33643 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 03:22:28 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:28 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:28.760260 19566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:28.750912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.129:51361)
0924 03:22:28.751487 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:28.751504 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:28.751628 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:28.752022 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:28.752027 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:28.752394 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:28.752585 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:28.752594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:28.753644 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:28.753649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:28.754590 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:28.754605 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:28.755468 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:28.755474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:28.755618 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:28.756268 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:28.756287 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:28.759902 (+ 3615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":288,"threads_started":1}
W20260924 03:22:28.760433 19556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:28.751047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:51361 (local address 127.10.12.190:36271)
0924 03:22:28.751210 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:28.751215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:28.751603 (+ 388us) server_negotiation.cc:407] Connection header received
0924 03:22:28.751727 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:28.751731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:28.751807 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:28.751984 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:28.752764 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:28.753495 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:28.754835 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:28.755320 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:28.755424 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:28.760234 (+ 4810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.129:51361: BlockingRecv error: recv got EOF from 127.10.12.129:51361 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 03:22:29 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:29 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:29.268947 19567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:29.257715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.130:41387)
0924 03:22:29.258462 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:29.258496 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:29.258692 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:29.259106 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:29.259111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:29.259595 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:29.259836 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:29.259849 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:29.261074 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:29.261078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:29.262323 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:29.262340 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:29.263276 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:29.263282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:29.263478 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:29.264271 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:29.264293 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:29.268551 (+ 4258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":371,"threads_started":1}
W20260924 03:22:29.269198 19556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:29.257867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:41387 (local address 127.10.12.190:36271)
0924 03:22:29.258110 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:29.258118 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:29.258602 (+ 484us) server_negotiation.cc:407] Connection header received
0924 03:22:29.258857 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:29.258861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:29.258940 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:29.259034 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:29.260044 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:29.260900 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:29.262474 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:29.263098 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:29.263220 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:29.268901 (+ 5681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.130:41387: BlockingRecv error: recv got EOF from 127.10.12.130:41387 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 03:22:29 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:29 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:29.729041 19568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:29.718070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.131:49245)
0924 03:22:29.718714 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:29.718742 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:29.718906 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:29.719340 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:29.719345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:29.719770 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:29.720004 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:29.720014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:29.721308 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:29.721315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:29.722541 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:29.722559 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:29.723621 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:29.723627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:29.723795 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:29.724513 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:29.724534 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:29.728609 (+ 4075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":335,"threads_started":1}
W20260924 03:22:29.729209 19556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:29.718254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:49245 (local address 127.10.12.190:36271)
0924 03:22:29.718484 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:29.718489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:29.718828 (+ 339us) server_negotiation.cc:407] Connection header received
0924 03:22:29.719066 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:29.719071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:29.719148 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:29.719260 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:29.720169 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:29.721116 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:29.722721 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:29.723478 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:29.723587 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:29.728972 (+ 5385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.131:49245: BlockingRecv error: recv got EOF from 127.10.12.131:49245 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 24 03:22:29 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:29 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:29.772264 19569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:29.761530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.129:43775)
0924 03:22:29.762165 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:29.762183 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:29.762341 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:29.762750 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:29.762755 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:29.763218 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:29.763415 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:29.763425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:29.764709 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:29.764716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:29.765804 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:29.765818 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:29.766544 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:29.766550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:29.766745 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:29.767458 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:29.767479 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:29.771901 (+ 4422us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":345,"threads_started":1}
W20260924 03:22:29.772552 19556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:29.761613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:43775 (local address 127.10.12.190:36271)
0924 03:22:29.761879 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:29.761886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:29.762271 (+ 385us) server_negotiation.cc:407] Connection header received
0924 03:22:29.762500 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:29.762505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:29.762577 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:29.762658 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:29.763576 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:29.764529 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:29.765967 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:29.766394 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:29.766505 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:29.772283 (+ 5778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.129:43775: BlockingRecv error: recv got EOF from 127.10.12.129:43775 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 03:22:30 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:30 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:30.280401 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:30.270334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.130:43157)
0924 03:22:30.271103 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:30.271120 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:30.271259 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:30.271602 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:30.271607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:30.272117 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:30.272310 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:30.272320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:30.273699 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:30.273703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:30.274724 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:30.274737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:30.275487 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:30.275493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:30.275681 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:30.276341 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:30.276360 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:30.280061 (+ 3701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":490,"threads_started":1}
W20260924 03:22:30.280560 19556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:30.270350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:43157 (local address 127.10.12.190:36271)
0924 03:22:30.270560 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:30.270567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:30.271207 (+ 640us) server_negotiation.cc:407] Connection header received
0924 03:22:30.271375 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:30.271382 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:30.271454 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:30.271535 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:30.272465 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:30.273551 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:30.274884 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:30.275338 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:30.275452 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:30.280306 (+ 4854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.130:43157: BlockingRecv error: recv got EOF from 127.10.12.130:43157 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 03:22:30 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:30 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:30.740365 19571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:30.730461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.131:58951)
0924 03:22:30.730954 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:30.730971 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:30.731110 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:30.731506 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:30.731510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:30.731922 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:30.732121 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:30.732130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:30.733133 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:30.733137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:30.734133 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:30.734146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:30.734869 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:30.734874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:30.735012 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:30.735662 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:30.735680 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:30.739986 (+ 4306us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":251,"threads_started":1}
W20260924 03:22:30.740521 19556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:30.730549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:58951 (local address 127.10.12.190:36271)
0924 03:22:30.730775 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:30.730782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:30.731060 (+ 278us) server_negotiation.cc:407] Connection header received
0924 03:22:30.731273 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:30.731277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:30.731343 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:30.731438 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:30.732251 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:30.732980 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:30.734248 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:30.734663 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:30.734757 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:30.740284 (+ 5527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.131:58951: BlockingRecv error: recv got EOF from 127.10.12.131:58951 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 24 03:22:30 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:30 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:30.782687 19572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:30.773400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.129:58367)
0924 03:22:30.774032 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:30.774049 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:30.774192 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:30.774541 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:30.774544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:30.775054 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:30.775226 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:30.775237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:30.776477 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:30.776482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:30.777576 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:30.777592 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:30.778296 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:30.778301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:30.778433 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:30.779046 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:30.779065 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:30.782436 (+ 3371us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":296,"threads_started":1}
W20260924 03:22:30.782936 19556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:30.773572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:58367 (local address 127.10.12.190:36271)
0924 03:22:30.773818 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:30.773825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:30.774141 (+ 316us) server_negotiation.cc:407] Connection header received
0924 03:22:30.774294 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:30.774298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:30.774364 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:30.774455 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:30.775390 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:30.776320 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:30.777723 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:30.778132 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:30.778229 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:30.782686 (+ 4457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.129:58367: BlockingRecv error: recv got EOF from 127.10.12.129:58367 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 24 03:22:31 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:31 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:31.295468 19573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:31.281946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.130:60343)
0924 03:22:31.282689 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:31.282711 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:31.282878 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:31.283207 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:31.283212 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:31.283660 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:31.283812 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:31.283822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:31.284893 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:31.284898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:31.286038 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:31.286055 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:31.286885 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:31.286891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:31.287073 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:31.291442 (+ 4369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:31.291467 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:31.295228 (+ 3761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":473,"threads_started":1}
W20260924 03:22:31.295746 19556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:31.282421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:60343 (local address 127.10.12.190:36271)
0924 03:22:31.282669 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:31.282676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:31.282806 (+ 130us) server_negotiation.cc:407] Connection header received
0924 03:22:31.283007 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:31.283011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:31.283076 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:31.283157 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:31.283996 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:31.284752 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:31.286186 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:31.286736 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:31.286852 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:31.295508 (+ 8656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.130:60343: BlockingRecv error: recv got EOF from 127.10.12.130:60343 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 24 03:22:31 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:31 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:31.750639 19574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:31.741607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.131:56157)
0924 03:22:31.742089 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:31.742104 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:31.742217 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:31.742470 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:31.742472 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:31.742879 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:31.743034 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:31.743043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:31.744105 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:31.744109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:31.745058 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:31.745072 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:31.745815 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:31.745820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:31.745974 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:31.746627 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:31.746645 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:31.750365 (+ 3720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":254,"threads_started":1}
W20260924 03:22:31.750851 19556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:31.741644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:56157 (local address 127.10.12.190:36271)
0924 03:22:31.741838 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:31.741845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:31.742205 (+ 360us) server_negotiation.cc:407] Connection header received
0924 03:22:31.742304 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:31.742307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:31.742369 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:31.742457 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:31.743188 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:31.743977 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:31.745227 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:31.745664 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:31.745766 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:31.750624 (+ 4858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.131:56157: BlockingRecv error: recv got EOF from 127.10.12.131:56157 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 03:22:31 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:31 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:31.793434 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:31.783711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.129:46681)
0924 03:22:31.784289 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:31.784308 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:31.784443 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:31.784796 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:31.784799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:31.785222 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:31.785392 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:31.785402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:31.786715 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:31.786720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:31.787691 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:31.787704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:31.788505 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:31.788510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:31.788658 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:31.789364 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:31.789383 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:31.793178 (+ 3795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":364,"threads_started":1}
W20260924 03:22:31.793871 19556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:31.783861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:46681 (local address 127.10.12.190:36271)
0924 03:22:31.784084 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:31.784088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:31.784412 (+ 324us) server_negotiation.cc:407] Connection header received
0924 03:22:31.784540 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:31.784544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:31.784618 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:31.784704 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:31.785740 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:31.786541 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:31.787845 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:31.788325 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:31.788426 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:31.793470 (+ 5044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.129:46681: BlockingRecv error: recv got EOF from 127.10.12.129:46681 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 03:22:32 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:32 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:32.306453 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:32.296633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.130:43633)
0924 03:22:32.297143 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:32.297161 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:32.297281 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:32.297761 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:32.297764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:32.298232 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:32.298405 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:32.298416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:32.299470 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:32.299474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:32.300635 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:32.300651 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:32.301461 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:32.301467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:32.301649 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:32.302497 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:32.302515 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:32.306130 (+ 3615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":260,"threads_started":1}
W20260924 03:22:32.306712 19577 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:32.296690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:43633 (local address 127.10.12.190:36271)
0924 03:22:32.297448 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:32.297452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:32.297468 (+ 16us) server_negotiation.cc:407] Connection header received
0924 03:22:32.297519 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:32.297522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:32.297582 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:32.297675 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:32.298575 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:32.299303 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:32.300789 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:32.301252 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:32.301364 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:32.306431 (+ 5067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.130:43633: BlockingRecv error: recv got EOF from 127.10.12.130:43633 (error 108)
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":563,"threads_started":1}
Sep 24 03:22:32 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:32 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:32.761245 19578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:32.751793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.131:40189)
0924 03:22:32.752287 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:32.752307 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:32.752447 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:32.752850 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:32.752856 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:32.753279 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:32.753519 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:32.753534 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:32.754694 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:32.754698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:32.755644 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:32.755658 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:32.756438 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:32.756443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:32.756580 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:32.757173 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:32.757194 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:32.760985 (+ 3791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":275,"threads_started":1}
W20260924 03:22:32.761371 19577 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:32.751994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:40189 (local address 127.10.12.190:36271)
0924 03:22:32.752192 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:32.752198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:32.752400 (+ 202us) server_negotiation.cc:407] Connection header received
0924 03:22:32.752605 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:32.752609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:32.752678 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:32.752775 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:32.753692 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:32.754557 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:32.755782 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:32.756265 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:32.756356 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:32.761179 (+ 4823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.131:40189: BlockingRecv error: recv got EOF from 127.10.12.131:40189 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 24 03:22:32 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:32 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:32.803980 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:32.794595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.129:57571)
0924 03:22:32.795178 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:32.795194 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:32.795315 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:32.795615 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:32.795618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:32.796081 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:32.796259 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:32.796268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:32.797318 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:32.797323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:32.798296 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:32.798309 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:32.799055 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:32.799061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:32.799178 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:32.799804 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:32.799821 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:32.803606 (+ 3785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":253,"threads_started":1}
W20260924 03:22:32.804109 19577 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:32.794632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:57571 (local address 127.10.12.190:36271)
0924 03:22:32.794812 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:32.794817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:32.795286 (+ 469us) server_negotiation.cc:407] Connection header received
0924 03:22:32.795410 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:32.795414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:32.795499 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:32.795587 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:32.796382 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:32.797165 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:32.798470 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:32.798896 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:32.798997 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:32.803826 (+ 4829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.129:57571: BlockingRecv error: recv got EOF from 127.10.12.129:57571 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 24 03:22:33 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:33 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:33.318691 19581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:33.307717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.130:47165)
0924 03:22:33.308448 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:33.308467 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:33.308623 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:33.309094 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:33.309098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:33.309577 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:33.309785 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:33.309796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:33.311202 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:33.311218 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:33.312542 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:33.312563 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:33.313399 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:33.313406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:33.313580 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:33.314283 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:33.314305 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:33.318273 (+ 3968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":374,"threads_started":1}
W20260924 03:22:33.318907 19580 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:33.307805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:47165 (local address 127.10.12.190:36271)
0924 03:22:33.308397 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:33.308403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:33.308554 (+ 151us) server_negotiation.cc:407] Connection header received
0924 03:22:33.308771 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:33.308775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:33.308868 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:33.308983 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:33.309980 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:33.311004 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:33.312704 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:33.313219 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:33.313342 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:33.318593 (+ 5251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.130:47165: BlockingRecv error: recv got EOF from 127.10.12.130:47165 (error 108)
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":328,"threads_started":1}
Sep 24 03:22:33 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:33 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:33.774127 19582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:33.762462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.131:56703)
0924 03:22:33.763207 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:33.763226 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:33.763380 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:33.763734 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:33.763737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:33.764424 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:33.764728 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:33.764743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:33.766080 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:33.766085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:33.767199 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:33.767216 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:33.768137 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:33.768144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:33.768332 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:33.769065 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:33.769087 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:33.773830 (+ 4743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":378,"threads_started":1}
W20260924 03:22:33.774281 19580 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:33.762592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:56703 (local address 127.10.12.190:36271)
0924 03:22:33.762816 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:33.762821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:33.763331 (+ 510us) server_negotiation.cc:407] Connection header received
0924 03:22:33.763502 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:33.763506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:33.763580 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:33.763688 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:33.764949 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:33.765905 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:33.767457 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:33.767920 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:33.768086 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:33.774069 (+ 5983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.131:56703: BlockingRecv error: recv got EOF from 127.10.12.131:56703 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 03:22:33 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:33 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:33.815598 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:33.805343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.129:38589)
0924 03:22:33.806034 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:33.806051 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:33.806221 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:33.806654 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:33.806659 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:33.807174 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:33.807378 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:33.807390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:33.808635 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:33.808641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:33.809684 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:33.809698 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:33.810562 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:33.810569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:33.810757 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:33.811476 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:33.811495 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:33.815266 (+ 3771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":339,"threads_started":1}
W20260924 03:22:33.815784 19580 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:33.805587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:38589 (local address 127.10.12.190:36271)
0924 03:22:33.805883 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:33.805891 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:33.806151 (+ 260us) server_negotiation.cc:407] Connection header received
0924 03:22:33.806396 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:33.806401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:33.806492 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:33.806589 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:33.807564 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:33.808470 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:33.809852 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:33.810328 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:33.810447 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:33.815518 (+ 5071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.129:38589: BlockingRecv error: recv got EOF from 127.10.12.129:38589 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Sep 24 03:22:34 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:34 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:34.331692 19584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:34.320409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.130:48305)
0924 03:22:34.321178 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:34.321203 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:34.321388 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:34.321950 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:34.321955 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:34.322526 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:34.322767 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:34.322778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:34.324264 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:34.324269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:34.325325 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:34.325340 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:34.326190 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:34.326197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:34.326332 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:34.327044 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:34.327063 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:34.331310 (+ 4247us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":375,"threads_started":1}
W20260924 03:22:34.332150 19585 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:34.320577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:48305 (local address 127.10.12.190:36271)
0924 03:22:34.321463 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:34.321469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:34.321493 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:22:34.321580 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:34.321586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:34.321714 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:34.321847 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:34.322929 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:34.324074 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:34.325497 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:34.326012 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:34.326141 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:34.331649 (+ 5508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.130:48305: BlockingRecv error: recv got EOF from 127.10.12.130:48305 (error 108)
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":638,"threads_started":1}
Sep 24 03:22:34 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:34 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:34.786517 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:34.775485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.131:51289)
0924 03:22:34.776286 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:34.776304 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:34.776491 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:34.776859 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:34.776864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:34.777390 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:34.777616 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:34.777627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:34.778841 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:34.778853 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:34.779902 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:34.779918 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:34.780899 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:34.780906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:34.781074 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:34.781804 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:34.781825 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:34.786192 (+ 4367us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":508,"threads_started":1}
W20260924 03:22:34.786762 19585 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:34.775686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:51289 (local address 127.10.12.190:36271)
0924 03:22:34.776020 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:34.776028 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:34.776440 (+ 412us) server_negotiation.cc:407] Connection header received
0924 03:22:34.776630 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:34.776635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:34.776708 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:34.776799 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:34.777790 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:34.778682 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:34.780127 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:34.780668 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:34.780793 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:34.786519 (+ 5726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.131:51289: BlockingRecv error: recv got EOF from 127.10.12.131:51289 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 24 03:22:34 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:34 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:34.828572 19587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:34.816951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.129:34577)
0924 03:22:34.817682 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:34.817700 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:34.817893 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:34.818297 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:34.818303 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:34.818824 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:34.819003 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:34.819014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:34.820215 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:34.820219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:34.821267 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:34.821284 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:34.822515 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:34.822524 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:34.822778 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:34.823547 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:34.823574 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:34.828233 (+ 4659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":473,"threads_started":1}
W20260924 03:22:34.829038 19585 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:34.817135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:34577 (local address 127.10.12.190:36271)
0924 03:22:34.817432 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:34.817439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:34.817803 (+ 364us) server_negotiation.cc:407] Connection header received
0924 03:22:34.818050 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:34.818054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:34.818147 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:34.818239 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:34.819164 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:34.820068 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:34.821491 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:34.822242 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:34.822394 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:34.828548 (+ 6154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.129:34577: BlockingRecv error: recv got EOF from 127.10.12.129:34577 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Sep 24 03:22:35 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:35 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:35.343297 19588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:35.333031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.130:54887)
0924 03:22:35.333623 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:35.333641 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:35.333766 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:35.334260 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:35.334263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:35.334847 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:35.335046 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:35.335057 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:35.336225 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:35.336230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:35.337209 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:35.337222 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:35.337984 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:35.337990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:35.338159 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:35.338740 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:35.338760 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:35.343009 (+ 4249us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":270,"threads_started":1}
W20260924 03:22:35.343554 19589 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:35.333157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:54887 (local address 127.10.12.190:36271)
0924 03:22:35.333918 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:35.333923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:35.333940 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:22:35.333998 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:35.334001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:35.334071 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:35.334166 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:35.335205 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:35.336089 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:35.337372 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:35.337834 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:35.337956 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:35.343256 (+ 5300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.130:54887: BlockingRecv error: recv got EOF from 127.10.12.130:54887 (error 108)
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":564,"threads_started":1}
Sep 24 03:22:35 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:35 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:35.798241 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:35.787867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.131:46903)
0924 03:22:35.788542 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:35.788560 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:35.788714 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:35.789139 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:35.789143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:35.789566 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:35.789730 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:35.789739 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:35.790813 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:35.790817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:35.791839 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:35.791853 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:35.792555 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:35.792560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:35.792688 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:35.793268 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:35.793287 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:35.797921 (+ 4634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":299,"threads_started":1}
W20260924 03:22:35.798455 19589 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:35.788101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:46903 (local address 127.10.12.190:36271)
0924 03:22:35.788420 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:35.788425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:35.788659 (+ 234us) server_negotiation.cc:407] Connection header received
0924 03:22:35.788835 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:35.788840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:35.788922 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:35.789020 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:35.789882 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:35.790654 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:35.791995 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:35.792407 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:35.792503 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:35.798231 (+ 5728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.131:46903: BlockingRecv error: recv got EOF from 127.10.12.131:46903 (error 108)
Metrics: {"server-negotiator.queue_time_us":198}
Sep 24 03:22:35 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:35 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:35.840827 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:35.829706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.129:48999)
0924 03:22:35.830308 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:35.830325 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:35.830469 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:35.831069 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:35.831074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:35.831522 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:35.831714 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:35.831725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:35.833235 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:35.833240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:35.834328 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:35.834344 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:35.835218 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:35.835223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:35.835363 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:35.836076 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:35.836098 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:35.840567 (+ 4469us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":326,"threads_started":1}
W20260924 03:22:35.841130 19589 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:35.829873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:48999 (local address 127.10.12.190:36271)
0924 03:22:35.830265 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:35.830270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:35.830409 (+ 139us) server_negotiation.cc:407] Connection header received
0924 03:22:35.830713 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:35.830719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:35.830821 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:35.830975 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:35.831945 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:35.833044 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:35.834532 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:35.835004 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:35.835145 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:35.840787 (+ 5642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.129:48999: BlockingRecv error: recv got EOF from 127.10.12.129:48999 (error 108)
Metrics: {"server-negotiator.queue_time_us":247}
Sep 24 03:22:36 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:36 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:36.360824 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:36.344440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.130:35447)
0924 03:22:36.344965 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:36.344982 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:36.345095 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:36.345549 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:36.345552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:36.345980 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:36.346157 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:36.346167 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:36.347440 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:36.347445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:36.348617 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:36.348634 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:36.349518 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:36.349525 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:36.349694 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:36.355652 (+ 5958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:36.355676 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:36.360277 (+ 4601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":262,"threads_started":1}
W20260924 03:22:36.361055 19593 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:36.344535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:35447 (local address 127.10.12.190:36271)
0924 03:22:36.345215 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:36.345220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:36.345237 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:22:36.345291 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:36.345295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:36.345369 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:36.345483 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:36.346346 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:36.347243 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:36.348800 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:36.349329 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:36.349433 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:36.360791 (+ 11358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.130:35447: BlockingRecv error: recv got EOF from 127.10.12.130:35447 (error 108)
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":492,"threads_started":1}
Sep 24 03:22:36 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:36 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:36.808874 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:36.799345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.131:53945)
0924 03:22:36.799991 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:36.800009 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:36.800139 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:36.800491 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:36.800494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:36.800867 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:36.801040 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:36.801050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:36.802294 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:36.802299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:36.803493 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:36.803509 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:36.804354 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:36.804360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:36.804491 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:36.805171 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:36.805190 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:36.808599 (+ 3409us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":251,"threads_started":1}
W20260924 03:22:36.808996 19593 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:36.799614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:53945 (local address 127.10.12.190:36271)
0924 03:22:36.799880 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:36.799885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:36.800092 (+ 207us) server_negotiation.cc:407] Connection header received
0924 03:22:36.800228 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:36.800231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:36.800293 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:36.800378 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:36.801253 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:36.802120 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:36.803658 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:36.804174 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:36.804278 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:36.808815 (+ 4537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.131:53945: BlockingRecv error: recv got EOF from 127.10.12.131:53945 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
Sep 24 03:22:36 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:36 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:36.853191 19595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:36.841792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.129:35113)
0924 03:22:36.842358 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:36.842377 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:36.842496 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:36.842878 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:36.842882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:36.843352 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:36.843542 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:36.843552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:36.844688 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:36.844692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:36.845775 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:36.845809 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:36.846683 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:36.846688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:36.846845 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:36.847755 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:36.847795 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:36.852799 (+ 5004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":326,"threads_started":1}
W20260924 03:22:36.853415 19593 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:36.841921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:35113 (local address 127.10.12.190:36271)
0924 03:22:36.842140 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:36.842147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:36.842456 (+ 309us) server_negotiation.cc:407] Connection header received
0924 03:22:36.842623 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:36.842627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:36.842704 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:36.842787 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:36.843693 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:36.844538 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:36.846014 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:36.846498 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:36.846617 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:36.853148 (+ 6531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.129:35113: BlockingRecv error: recv got EOF from 127.10.12.129:35113 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 03:22:37 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:37 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:37.373065 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:37.361841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.130:34611)
0924 03:22:37.362523 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:37.362549 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:37.362722 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:37.363683 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:37.363688 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:37.364409 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:37.364618 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:37.364632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:37.366267 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:37.366272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:37.367198 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:37.367216 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:37.368227 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:37.368232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:37.368379 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:37.369035 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:37.369054 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:37.372751 (+ 3697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":311,"threads_started":1}
W20260924 03:22:37.373242 19597 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:37.362162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:34611 (local address 127.10.12.190:36271)
0924 03:22:37.363296 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:37.363302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:37.363321 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:22:37.363372 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:37.363376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:37.363439 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:37.363550 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:37.364868 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:37.366011 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:37.367475 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:37.368010 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:37.368144 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:37.372974 (+ 4830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.130:34611: BlockingRecv error: recv got EOF from 127.10.12.130:34611 (error 108)
Metrics: {"server-negotiator.queue_time_us":1023,"thread_start_us":911,"threads_started":1}
Sep 24 03:22:37 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:37 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:37.819561 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:37.810147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.131:32853)
0924 03:22:37.810692 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:37.810709 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:37.810850 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:37.811138 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:37.811141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:37.811565 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:37.811750 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:37.811760 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:37.812833 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:37.812837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:37.813812 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:37.813826 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:37.814623 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:37.814628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:37.814766 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:37.815420 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:37.815439 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:37.819300 (+ 3861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":332,"threads_started":1}
W20260924 03:22:37.819700 19597 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:37.810320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:32853 (local address 127.10.12.190:36271)
0924 03:22:37.810490 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:37.810496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:37.810866 (+ 370us) server_negotiation.cc:407] Connection header received
0924 03:22:37.810942 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:37.810946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:37.811011 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:37.811109 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:37.811939 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:37.812683 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:37.813987 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:37.814436 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:37.814540 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:37.819514 (+ 4974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.131:32853: BlockingRecv error: recv got EOF from 127.10.12.131:32853 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 03:22:37 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:37 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:37.864266 19599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:37.854490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.129:47013)
0924 03:22:37.855069 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:37.855086 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:37.855216 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:37.855574 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:37.855578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:37.856220 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:37.856471 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:37.856485 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:37.857731 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:37.857736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:37.858795 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:37.858811 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:37.859640 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:37.859645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:37.859799 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:37.860473 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:37.860493 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:37.864006 (+ 3513us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":307,"threads_started":1}
W20260924 03:22:37.864611 19597 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:37.854642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:47013 (local address 127.10.12.190:36271)
0924 03:22:37.854844 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:37.854851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:37.855169 (+ 318us) server_negotiation.cc:407] Connection header received
0924 03:22:37.855340 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:37.855345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:37.855433 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:37.855521 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:37.856623 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:37.857574 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:37.858947 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:37.859440 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:37.859546 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:37.864250 (+ 4704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.129:47013: BlockingRecv error: recv got EOF from 127.10.12.129:47013 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 03:22:38 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:38 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:38.384984 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:38.374127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.130:38953)
0924 03:22:38.374699 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:38.374717 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:38.374846 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:38.375240 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:38.375244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:38.375798 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:38.376037 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:38.376049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:38.377221 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:38.377225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:38.378234 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:38.378248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:38.379099 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:38.379104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:38.379255 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:38.379881 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:38.379899 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:38.384630 (+ 4731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":286,"threads_started":1}
W20260924 03:22:38.385452 19601 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:38.374373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:38953 (local address 127.10.12.190:36271)
0924 03:22:38.374960 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:38.374965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:38.374982 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:22:38.375041 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:38.375044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:38.375103 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:38.375190 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:38.376213 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:38.377061 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:38.378396 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:38.378848 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:38.378967 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:38.385015 (+ 6048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.130:38953: BlockingRecv error: recv got EOF from 127.10.12.130:38953 (error 108)
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":387,"threads_started":1}
Sep 24 03:22:38 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:38 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:38.830186 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:38.820722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.131:41511)
0924 03:22:38.821340 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:38.821356 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:38.821499 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:38.821802 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:38.821806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:38.822246 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:38.822413 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:38.822422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:38.823470 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:38.823475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:38.824506 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:38.824519 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:38.825301 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:38.825315 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:38.825604 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:38.826213 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:38.826231 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:38.829788 (+ 3557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":385,"threads_started":1}
W20260924 03:22:38.830268 19601 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:38.820843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:41511 (local address 127.10.12.190:36271)
0924 03:22:38.821076 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:38.821083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:38.821440 (+ 357us) server_negotiation.cc:407] Connection header received
0924 03:22:38.821612 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:38.821616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:38.821677 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:38.821755 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:38.822587 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:38.823317 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:38.824653 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:38.825056 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:38.825169 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:38.830051 (+ 4882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.131:41511: BlockingRecv error: recv got EOF from 127.10.12.131:41511 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 24 03:22:38 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:38 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:38.874873 19603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:38.865355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.129:53333)
0924 03:22:38.865847 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:38.865865 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:38.866001 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:38.866304 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:38.866307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:38.866733 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:38.866888 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:38.866900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:38.868202 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:38.868206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:38.869447 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:38.869462 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:38.870211 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:38.870216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:38.870356 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:38.870969 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:38.870987 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:38.874532 (+ 3545us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":259,"threads_started":1}
W20260924 03:22:38.875023 19601 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:38.865556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:53333 (local address 127.10.12.190:36271)
0924 03:22:38.865733 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:38.865738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:38.866011 (+ 273us) server_negotiation.cc:407] Connection header received
0924 03:22:38.866108 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:38.866112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:38.866189 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:38.866268 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:38.867050 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:38.868052 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:38.869619 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:38.870052 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:38.870149 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:38.874778 (+ 4629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.129:53333: BlockingRecv error: recv got EOF from 127.10.12.129:53333 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 03:22:39 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:39 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:39.397547 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:39.386422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.130:60269)
0924 03:22:39.387100 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:39.387125 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:39.387305 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:39.387712 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:39.387717 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:39.388370 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:39.388568 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:39.388579 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:39.389790 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:39.389794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:39.390760 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:39.390774 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:39.391580 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:39.391584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:39.391740 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:39.392650 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:39.392673 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:39.397236 (+ 4563us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":328,"threads_started":1}
W20260924 03:22:39.397832 19605 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:39.386593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:60269 (local address 127.10.12.190:36271)
0924 03:22:39.387304 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:39.387310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:39.387328 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:22:39.387436 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:39.387440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:39.387525 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:39.387660 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:39.388718 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:39.389594 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:39.390937 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:39.391427 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:39.391536 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:39.397548 (+ 6012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.130:60269: BlockingRecv error: recv got EOF from 127.10.12.130:60269 (error 108)
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":474,"threads_started":1}
Sep 24 03:22:39 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:39 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.131@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:39.843084 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:39.831580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.131:42401)
0924 03:22:39.832171 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:39.832189 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:39.832341 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:39.832737 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:39.832740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:39.833247 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:39.833424 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:39.833434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:39.834714 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:39.834719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:39.835855 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:39.835874 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:39.836993 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:39.836998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:39.837181 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:39.838040 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:39.838064 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:39.842795 (+ 4731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":295,"threads_started":1}
W20260924 03:22:39.843387 19605 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:39.831766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:42401 (local address 127.10.12.190:36271)
0924 03:22:39.832034 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:39.832040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:39.832312 (+ 272us) server_negotiation.cc:407] Connection header received
0924 03:22:39.832460 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:39.832464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:39.832542 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:39.832643 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:39.833595 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:39.834519 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:39.836160 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:39.836822 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:39.836947 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:39.843116 (+ 6169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.131:42401: BlockingRecv error: recv got EOF from 127.10.12.131:42401 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 24 03:22:39 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:39 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.129@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:39.887606 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:39.876414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.129:58557)
0924 03:22:39.877060 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:39.877083 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:39.877251 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:39.877770 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:39.877774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:39.878210 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:39.878822 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:39.878842 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:39.880090 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:39.880151 (+ 61us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:39.881235 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:39.881250 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:39.882001 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:39.882006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:39.882166 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:39.882929 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:39.882951 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:39.887242 (+ 4291us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":334,"threads_started":1}
W20260924 03:22:39.887876 19605 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:39.876501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:58557 (local address 127.10.12.190:36271)
0924 03:22:39.876888 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:39.876897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:39.877239 (+ 342us) server_negotiation.cc:407] Connection header received
0924 03:22:39.877468 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:39.877472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:39.877565 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:39.877656 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:39.879043 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:39.879906 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:39.881402 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:39.881845 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:39.881945 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:39.887570 (+ 5625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.129:58557: BlockingRecv error: recv got EOF from 127.10.12.129:58557 (error 108)
Metrics: {"server-negotiator.queue_time_us":184}
Sep 24 03:22:40 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:22:40 dist-test-slave-00qs krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.12.130@KRBTEST.COM for oryx/127.10.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:22:40.410174 19608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:40.399100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36271 (local address 127.10.12.130:53421)
0924 03:22:40.399580 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:40.399597 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:40.399713 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:40.400259 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:40.400264 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:40.400817 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:40.401110 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:40.401122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:40.402255 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:40.402259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:40.403335 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:40.403351 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:40.404339 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:40.404345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:40.404540 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:40.405394 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:40.405421 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:40.409872 (+ 4451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.12.190:36271: Server oryx/127.10.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":224,"threads_started":1}
W20260924 03:22:40.410369 19609 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:40.399291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:53421 (local address 127.10.12.190:36271)
0924 03:22:40.399848 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:40.399853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:40.399871 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:22:40.399940 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:40.399943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:40.400060 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:40.400162 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:40.401286 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:40.402099 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:40.403529 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:40.404183 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:40.404293 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:40.410133 (+ 5840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.12.130:53421: BlockingRecv error: recv got EOF from 127.10.12.130:53421 (error 108)
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":337,"threads_started":1}
I20260924 03:22:40.610633 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 19139
I20260924 03:22:40.623868 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:40.611985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34818 (local address 127.10.12.129:34843)
0924 03:22:40.612684 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:40.612692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:40.612723 (+ 31us) server_negotiation.cc:407] Connection header received
0924 03:22:40.612803 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:40.612809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:40.613070 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:40.613250 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:40.614743 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:40.615810 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:40.617161 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:40.617754 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:40.617873 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:40.620598 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:40.620608 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:40.620623 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:40.620672 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:40.623317 (+ 2645us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.12.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 03:22:40.623641 (+ 324us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34818: Request ticket server kudu/127.10.12.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":358,"threads_started":1}
W20260924 03:22:40.623991 19611 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34818: Request ticket server kudu/127.10.12.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 03:22:40.624140 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 19139: Not authorized: Client connection negotiation failed: client connection to 127.10.12.129:34843: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.12.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 03:22:40.624361 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 19139
I20260924 03:22:40.638444 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 19290
I20260924 03:22:40.654122 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:40.639630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56462 (local address 127.10.12.130:43769)
0924 03:22:40.640360 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:40.640369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:40.640406 (+ 37us) server_negotiation.cc:407] Connection header received
0924 03:22:40.640499 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:40.640508 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:40.640837 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:40.641074 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:40.642870 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:40.644174 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:40.645722 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:40.646578 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:40.646711 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:40.650266 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:40.650277 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:40.650293 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:40.650350 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:40.653517 (+ 3167us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.12.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 03:22:40.653861 (+ 344us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56462: Request ticket server kudu/127.10.12.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":637,"threads_started":1}
W20260924 03:22:40.654228 19612 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56462: Request ticket server kudu/127.10.12.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 03:22:40.654351 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 19290: Not authorized: Client connection negotiation failed: client connection to 127.10.12.130:43769: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.12.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 03:22:40.654573 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 19290
I20260924 03:22:40.668236 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 19423
I20260924 03:22:40.682500 19613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:40.669478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35722 (local address 127.10.12.131:36177)
0924 03:22:40.670111 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:40.670120 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:40.670154 (+ 34us) server_negotiation.cc:407] Connection header received
0924 03:22:40.670238 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:40.670245 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:40.670501 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:40.670672 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:40.672467 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:40.673631 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:40.674974 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:40.675588 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:40.675703 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:40.678557 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:40.678567 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:40.678582 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:40.678629 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:40.681846 (+ 3217us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.12.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 03:22:40.682225 (+ 379us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35722: Request ticket server kudu/127.10.12.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":290,"threads_started":1}
W20260924 03:22:40.682622 19613 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35722: Request ticket server kudu/127.10.12.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 03:22:40.682753 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 19423: Not authorized: Client connection negotiation failed: client connection to 127.10.12.131:36177: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.12.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 03:22:40.682962 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 19423
I20260924 03:22:40.696209 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 19070
I20260924 03:22:40.697759 19132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:40.815830 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 19070
2026-09-24T03:22:40Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21339 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T03:22:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:22:40Z Disabled control of system clock
I20260924 03:22:40.891731 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:45917
--webserver_interface=127.10.12.190
--webserver_port=0
--builtin_ntp_servers=127.10.12.148:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:45917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 03:22:41.234375 19618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:41.234683 19618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:41.234771 19618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:41.242173 19618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:22:41.242304 19618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:41.242341 19618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:41.242389 19618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:22:41.242432 19618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:22:41.251847 19618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:45917
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:45917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:41.253180 19618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:41.255002 19618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:41.267987 19626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:41.267998 19623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:41.267920 19624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:41.268461 19618 server_base.cc:1034] running on GCE node
I20260924 03:22:41.269345 19618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:41.270941 19618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:41.272212 19618 hybrid_clock.cc:648] HybridClock initialized: now 1790220161272191 us; error 55 us; skew 500 ppm
I20260924 03:22:41.272696 19618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:22:41.275489 19618 webserver.cc:460] Webserver started at http://127.10.12.190:37429/ using document root <none> and password file <none>
I20260924 03:22:41.276310 19618 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:41.276435 19618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:41.276764 19618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:41.280916 19618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "c06e0a3aeacd49c9bd6d3ef85dd68597"
format_stamp: "Formatted at 2026-09-24 03:22:41 on dist-test-slave-00qs"
server_key: "ea7798414c98e6731a8be4761aedb3ec"
server_key_iv: "de14c647a428b49ee9b158b0b45d58b7"
server_key_version: "encryptionkey@0"
I20260924 03:22:41.281996 19618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "c06e0a3aeacd49c9bd6d3ef85dd68597"
format_stamp: "Formatted at 2026-09-24 03:22:41 on dist-test-slave-00qs"
server_key: "ea7798414c98e6731a8be4761aedb3ec"
server_key_iv: "de14c647a428b49ee9b158b0b45d58b7"
server_key_version: "encryptionkey@0"
I20260924 03:22:41.287994 19618 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:22:41.292374 19632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:41.294083 19618 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:22:41.294297 19618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "c06e0a3aeacd49c9bd6d3ef85dd68597"
format_stamp: "Formatted at 2026-09-24 03:22:41 on dist-test-slave-00qs"
server_key: "ea7798414c98e6731a8be4761aedb3ec"
server_key_iv: "de14c647a428b49ee9b158b0b45d58b7"
server_key_version: "encryptionkey@0"
I20260924 03:22:41.294571 19618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:41.314716 19618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:41.315680 19618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:41.316022 19618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:41.336267 19618 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:45917
I20260924 03:22:41.336282 19683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:45917 every 8 connection(s)
I20260924 03:22:41.337678 19618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:22:41.342834 19684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:41.346532 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 19618
I20260924 03:22:41.346771 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:22:41.347532 10290 external_mini_cluster.cc:1428] Setting key c05db26b66b2cc5930a1ce5c30c799c6
I20260924 03:22:41.355504 19684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c06e0a3aeacd49c9bd6d3ef85dd68597: Bootstrap starting.
I20260924 03:22:41.358893 19684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c06e0a3aeacd49c9bd6d3ef85dd68597: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:41.360177 19684 log.cc:824] T 00000000000000000000000000000000 P c06e0a3aeacd49c9bd6d3ef85dd68597: Log is configured to *not* fsync() on all Append() calls
I20260924 03:22:41.363291 19684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c06e0a3aeacd49c9bd6d3ef85dd68597: No bootstrap required, opened a new log
I20260924 03:22:41.367102 19687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:41.350254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58924 (local address 127.10.12.190:45917)
0924 03:22:41.351390 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:41.351402 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:41.351451 (+ 49us) server_negotiation.cc:407] Connection header received
0924 03:22:41.353442 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:41.353473 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:41.354027 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:41.354391 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:41.355191 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:41.356728 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:41.358106 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:41.358834 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:41.359029 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:41.364419 (+ 5390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:41.364430 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:41.364448 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:22:41.365067 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:41.365173 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:22:41.365203 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:41.365335 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:41.365614 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 03:22:41.366003 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":366,"threads_started":1}
I20260924 03:22:41.372682 19684 raft_consensus.cc:348] T 00000000000000000000000000000000 P c06e0a3aeacd49c9bd6d3ef85dd68597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c06e0a3aeacd49c9bd6d3ef85dd68597" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45917 } }
I20260924 03:22:41.373121 19684 raft_consensus.cc:374] T 00000000000000000000000000000000 P c06e0a3aeacd49c9bd6d3ef85dd68597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:41.373257 19684 raft_consensus.cc:729] T 00000000000000000000000000000000 P c06e0a3aeacd49c9bd6d3ef85dd68597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c06e0a3aeacd49c9bd6d3ef85dd68597, State: Initialized, Role: FOLLOWER
I20260924 03:22:41.373947 19684 consensus_queue.cc:260] T 00000000000000000000000000000000 P c06e0a3aeacd49c9bd6d3ef85dd68597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c06e0a3aeacd49c9bd6d3ef85dd68597" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45917 } }
I20260924 03:22:41.374185 19684 raft_consensus.cc:388] T 00000000000000000000000000000000 P c06e0a3aeacd49c9bd6d3ef85dd68597 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:22:41.374286 19684 raft_consensus.cc:482] T 00000000000000000000000000000000 P c06e0a3aeacd49c9bd6d3ef85dd68597 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:22:41.374414 19684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c06e0a3aeacd49c9bd6d3ef85dd68597 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:41.376745 19684 raft_consensus.cc:504] T 00000000000000000000000000000000 P c06e0a3aeacd49c9bd6d3ef85dd68597 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c06e0a3aeacd49c9bd6d3ef85dd68597" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45917 } }
I20260924 03:22:41.377280 19684 leader_election.cc:302] T 00000000000000000000000000000000 P c06e0a3aeacd49c9bd6d3ef85dd68597 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c06e0a3aeacd49c9bd6d3ef85dd68597; no voters:
I20260924 03:22:41.377929 19684 leader_election.cc:288] T 00000000000000000000000000000000 P c06e0a3aeacd49c9bd6d3ef85dd68597 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:22:41.378310 19689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c06e0a3aeacd49c9bd6d3ef85dd68597 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:22:41.379063 19689 raft_consensus.cc:686] T 00000000000000000000000000000000 P c06e0a3aeacd49c9bd6d3ef85dd68597 [term 1 LEADER]: Becoming Leader. State: Replica: c06e0a3aeacd49c9bd6d3ef85dd68597, State: Running, Role: LEADER
I20260924 03:22:41.379635 19689 consensus_queue.cc:237] T 00000000000000000000000000000000 P c06e0a3aeacd49c9bd6d3ef85dd68597 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c06e0a3aeacd49c9bd6d3ef85dd68597" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45917 } }
I20260924 03:22:41.380131 19684 sys_catalog.cc:564] T 00000000000000000000000000000000 P c06e0a3aeacd49c9bd6d3ef85dd68597 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:22:41.382937 19690 sys_catalog.cc:455] T 00000000000000000000000000000000 P c06e0a3aeacd49c9bd6d3ef85dd68597 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c06e0a3aeacd49c9bd6d3ef85dd68597" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c06e0a3aeacd49c9bd6d3ef85dd68597" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45917 } } }
I20260924 03:22:41.382964 19691 sys_catalog.cc:455] T 00000000000000000000000000000000 P c06e0a3aeacd49c9bd6d3ef85dd68597 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c06e0a3aeacd49c9bd6d3ef85dd68597. Latest consensus state: current_term: 1 leader_uuid: "c06e0a3aeacd49c9bd6d3ef85dd68597" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c06e0a3aeacd49c9bd6d3ef85dd68597" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45917 } } }
I20260924 03:22:41.383453 19691 sys_catalog.cc:458] T 00000000000000000000000000000000 P c06e0a3aeacd49c9bd6d3ef85dd68597 [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:41.383450 19690 sys_catalog.cc:458] T 00000000000000000000000000000000 P c06e0a3aeacd49c9bd6d3ef85dd68597 [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:41.384881 19697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:22:41.392192 19697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:22:41.402719 19697 catalog_manager.cc:1347] Generated new cluster ID: 409da137d62a467ca8edf71e3de1d9ba
I20260924 03:22:41.402864 19697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:22:41.432808 19697 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:22:41.434283 19697 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:22:41.445446 19697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c06e0a3aeacd49c9bd6d3ef85dd68597: Generated new TSK 0
I20260924 03:22:41.446828 19697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:22:41.455878 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:45917
--builtin_ntp_servers=127.10.12.148:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 03:22:41.800323 19708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:41.800642 19708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:41.800752 19708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:41.808348 19708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:41.808477 19708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:41.808585 19708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:22:41.818672 19708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:45917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:41.820083 19708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:41.821959 19708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:41.835866 19716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:41.835927 19714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:41.835983 19713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:41.836650 19708 server_base.cc:1034] running on GCE node
I20260924 03:22:41.837275 19708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:41.838323 19708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:41.839686 19708 hybrid_clock.cc:648] HybridClock initialized: now 1790220161839592 us; error 47 us; skew 500 ppm
I20260924 03:22:41.840369 19708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:22:41.849493 19708 webserver.cc:460] Webserver started at http://127.10.12.129:39353/ using document root <none> and password file <none>
I20260924 03:22:41.850315 19708 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:41.850433 19708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:41.850770 19708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:41.853981 19708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "0bbb2bda1047439e9195c6973f446228"
format_stamp: "Formatted at 2026-09-24 03:22:41 on dist-test-slave-00qs"
server_key: "2e2e527084a7e497ea5d5e965e07a0a5"
server_key_iv: "4b0957cdc92c1a4c3d62829ca8d18e38"
server_key_version: "encryptionkey@0"
I20260924 03:22:41.855033 19708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "0bbb2bda1047439e9195c6973f446228"
format_stamp: "Formatted at 2026-09-24 03:22:41 on dist-test-slave-00qs"
server_key: "2e2e527084a7e497ea5d5e965e07a0a5"
server_key_iv: "4b0957cdc92c1a4c3d62829ca8d18e38"
server_key_version: "encryptionkey@0"
I20260924 03:22:41.861234 19708 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260924 03:22:41.865640 19722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:41.867347 19708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 03:22:41.867568 19708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "0bbb2bda1047439e9195c6973f446228"
format_stamp: "Formatted at 2026-09-24 03:22:41 on dist-test-slave-00qs"
server_key: "2e2e527084a7e497ea5d5e965e07a0a5"
server_key_iv: "4b0957cdc92c1a4c3d62829ca8d18e38"
server_key_version: "encryptionkey@0"
I20260924 03:22:41.867857 19708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:41.883701 19708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:41.884676 19708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:41.884977 19708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:41.886047 19708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:22:41.887590 19708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:22:41.887691 19708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:41.887804 19708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:22:41.887873 19708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:41.922300 19708 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:35743
I20260924 03:22:41.922430 19834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:35743 every 8 connection(s)
I20260924 03:22:41.923846 19708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:22:41.931393 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 19708
I20260924 03:22:41.931594 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:22:41.932487 10290 external_mini_cluster.cc:1428] Setting key 0404785aae8dcebdc07774bc742d8a8f
I20260924 03:22:41.936225 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:45917
--builtin_ntp_servers=127.10.12.148:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 03:22:41.943274 19838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:41.927343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:45917 (local address 127.10.12.129:41501)
0924 03:22:41.928421 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:41.928462 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:41.930572 (+ 2110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:41.931838 (+ 1266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:41.931853 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:41.932007 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:41.932952 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:41.932984 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:41.934519 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:41.934525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:41.936746 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:41.936768 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:41.938629 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:41.938636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:41.938951 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:41.939783 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:22:41.939787 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:41.939869 (+ 82us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:22:41.940027 (+ 158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:41.940503 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:41.940516 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:41.940526 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 03:22:41.940767 (+ 241us) client_negotiation.cc:240] Negotiation successful
0924 03:22:41.941587 (+ 820us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":324,"threads_started":1}
I20260924 03:22:41.944464 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:41.926737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:41501 (local address 127.10.12.190:45917)
0924 03:22:41.927245 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:41.927250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:41.928603 (+ 1353us) server_negotiation.cc:407] Connection header received
0924 03:22:41.931070 (+ 2467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:41.931076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:41.931180 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:41.931290 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:41.933220 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:41.934289 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:41.936994 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:41.938333 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:41.938497 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:41.940293 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:41.940300 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:41.940304 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:22:41.940331 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:41.940384 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:22:41.940401 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:41.940477 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:41.943860 (+ 3383us) server_negotiation.cc:299] Negotiation successful
0924 03:22:41.944020 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":269,"threads_started":1}
I20260924 03:22:41.946154 19835 heartbeater.cc:347] Connected to a master server at 127.10.12.190:45917
I20260924 03:22:41.946746 19835 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:41.947994 19835 heartbeater.cc:511] Master 127.10.12.190:45917 requested a full tablet report, sending...
I20260924 03:22:41.950913 19649 ts_manager.cc:194] Registered new tserver with Master: 0bbb2bda1047439e9195c6973f446228 (127.10.12.129:35743)
I20260924 03:22:41.953017 19649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.12.129:41501
W20260924 03:22:42.295667 19839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:42.296018 19839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:42.296149 19839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:42.304029 19839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:42.304186 19839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:42.304301 19839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:22:42.313819 19839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:45917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:42.315174 19839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:42.317366 19839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:42.331511 19845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:42.331665 19847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:42.331825 19844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:42.332645 19839 server_base.cc:1034] running on GCE node
I20260924 03:22:42.333343 19839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:42.334452 19839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:42.335704 19839 hybrid_clock.cc:648] HybridClock initialized: now 1790220162335636 us; error 62 us; skew 500 ppm
I20260924 03:22:42.336148 19839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:22:42.339174 19839 webserver.cc:460] Webserver started at http://127.10.12.130:42109/ using document root <none> and password file <none>
I20260924 03:22:42.340144 19839 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:42.340267 19839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:42.340642 19839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:42.344681 19839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "ceb0bc7848814274a12cf14839f36251"
format_stamp: "Formatted at 2026-09-24 03:22:42 on dist-test-slave-00qs"
server_key: "8fd27e4283110b113c14742b1c20b956"
server_key_iv: "421f2ef520b2af1fd5275d4814772fb2"
server_key_version: "encryptionkey@0"
I20260924 03:22:42.345764 19839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "ceb0bc7848814274a12cf14839f36251"
format_stamp: "Formatted at 2026-09-24 03:22:42 on dist-test-slave-00qs"
server_key: "8fd27e4283110b113c14742b1c20b956"
server_key_iv: "421f2ef520b2af1fd5275d4814772fb2"
server_key_version: "encryptionkey@0"
I20260924 03:22:42.352810 19839 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260924 03:22:42.357430 19853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:42.359093 19839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260924 03:22:42.359287 19839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "ceb0bc7848814274a12cf14839f36251"
format_stamp: "Formatted at 2026-09-24 03:22:42 on dist-test-slave-00qs"
server_key: "8fd27e4283110b113c14742b1c20b956"
server_key_iv: "421f2ef520b2af1fd5275d4814772fb2"
server_key_version: "encryptionkey@0"
I20260924 03:22:42.359530 19839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:42.373911 19839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:42.374822 19839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:42.375141 19839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:42.376158 19839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:22:42.377724 19839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:22:42.377825 19839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:42.377948 19839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:22:42.378018 19839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:42.413033 19839 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:45077
I20260924 03:22:42.413079 19965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:45077 every 8 connection(s)
I20260924 03:22:42.414426 19839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:22:42.416677 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 19839
I20260924 03:22:42.416827 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:22:42.417304 10290 external_mini_cluster.cc:1428] Setting key a5f85468a93b213b163e5e01360a937c
I20260924 03:22:42.420380 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:45917
--builtin_ntp_servers=127.10.12.148:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 03:22:42.429224 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:42.417567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:45985 (local address 127.10.12.190:45917)
0924 03:22:42.417783 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:42.417790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:42.419556 (+ 1766us) server_negotiation.cc:407] Connection header received
0924 03:22:42.421726 (+ 2170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:42.421736 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:42.421966 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:42.422097 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:42.423390 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:42.424392 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:42.426148 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:42.426701 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:42.426837 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:42.428395 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:42.428401 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:42.428406 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:22:42.428429 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:42.428460 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:22:42.428472 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:42.428550 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:42.428794 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 03:22:42.428919 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:22:42.430474 19968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:42.418092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:45917 (local address 127.10.12.130:45985)
0924 03:22:42.419383 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:42.419455 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:42.421208 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:42.422378 (+ 1170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:42.422388 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:42.422529 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:42.423175 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:42.423194 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:42.424590 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:42.424594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:42.425976 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:42.425993 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:42.426972 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:42.426979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:42.427209 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:42.428037 (+ 828us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:22:42.428040 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:42.428088 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:22:42.428183 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:42.428568 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:42.428581 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:42.428590 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 03:22:42.428766 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 03:22:42.429222 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":377,"threads_started":1}
I20260924 03:22:42.432289 19966 heartbeater.cc:347] Connected to a master server at 127.10.12.190:45917
I20260924 03:22:42.432664 19966 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:42.433517 19966 heartbeater.cc:511] Master 127.10.12.190:45917 requested a full tablet report, sending...
I20260924 03:22:42.435261 19649 ts_manager.cc:194] Registered new tserver with Master: ceb0bc7848814274a12cf14839f36251 (127.10.12.130:45077)
I20260924 03:22:42.436408 19649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.12.130:45985
W20260924 03:22:42.760394 19969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:42.760697 19969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:42.760778 19969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:42.768386 19969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:42.768510 19969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:42.768616 19969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:22:42.777976 19969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:45917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:42.779214 19969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:42.781087 19969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:42.794770 19977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:42.794803 19974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:42.794736 19975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:42.795446 19969 server_base.cc:1034] running on GCE node
I20260924 03:22:42.796131 19969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:42.797163 19969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:42.798434 19969 hybrid_clock.cc:648] HybridClock initialized: now 1790220162798376 us; error 49 us; skew 500 ppm
I20260924 03:22:42.798787 19969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:22:42.801268 19969 webserver.cc:460] Webserver started at http://127.10.12.131:35517/ using document root <none> and password file <none>
I20260924 03:22:42.802018 19969 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:42.802111 19969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:42.802438 19969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:42.805521 19969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "cc7516968e6245bab986f2ef05422ecb"
format_stamp: "Formatted at 2026-09-24 03:22:42 on dist-test-slave-00qs"
server_key: "a5d454856f41b492fcdb692e7e8a1bd5"
server_key_iv: "6e1336944f8af35b3232e99dd1f9352c"
server_key_version: "encryptionkey@0"
I20260924 03:22:42.806353 19969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "cc7516968e6245bab986f2ef05422ecb"
format_stamp: "Formatted at 2026-09-24 03:22:42 on dist-test-slave-00qs"
server_key: "a5d454856f41b492fcdb692e7e8a1bd5"
server_key_iv: "6e1336944f8af35b3232e99dd1f9352c"
server_key_version: "encryptionkey@0"
I20260924 03:22:42.812153 19969 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:22:42.816085 19983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:42.817694 19969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:22:42.817890 19969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "cc7516968e6245bab986f2ef05422ecb"
format_stamp: "Formatted at 2026-09-24 03:22:42 on dist-test-slave-00qs"
server_key: "a5d454856f41b492fcdb692e7e8a1bd5"
server_key_iv: "6e1336944f8af35b3232e99dd1f9352c"
server_key_version: "encryptionkey@0"
I20260924 03:22:42.818120 19969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:42.838351 19969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:42.839427 19969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:42.839749 19969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:42.840902 19969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:22:42.842504 19969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:22:42.842597 19969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:42.842700 19969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:22:42.842772 19969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:42.874068 19969 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:43767
I20260924 03:22:42.874126 20095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:43767 every 8 connection(s)
I20260924 03:22:42.875414 19969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:22:42.876825 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 19969
I20260924 03:22:42.876976 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:22:42.877467 10290 external_mini_cluster.cc:1428] Setting key 8ffe7eaf456b9eb8d6f1430454a031ff
I20260924 03:22:42.888820 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:42.878244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:49995 (local address 127.10.12.190:45917)
0924 03:22:42.878478 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:42.878486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:42.880095 (+ 1609us) server_negotiation.cc:407] Connection header received
0924 03:22:42.882092 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:42.882098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:42.882191 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:42.882278 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:42.883449 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:42.884273 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:42.885948 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:42.886364 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:42.886478 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:42.887914 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:42.887920 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:42.887923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:22:42.887991 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:42.888032 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:22:42.888060 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:42.888145 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:42.888385 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 03:22:42.888493 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 03:22:42.890003 20098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:42.878868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:45917 (local address 127.10.12.131:49995)
0924 03:22:42.879905 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:42.879988 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:42.881792 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:42.882563 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:42.882573 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:42.882674 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:42.883270 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:42.883287 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:42.884431 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:42.884460 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:42.885769 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:42.885788 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:42.886608 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:42.886615 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:42.886857 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:42.887560 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:22:42.887563 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:42.887609 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:22:42.887701 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:42.888160 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:42.888173 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:42.888182 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 03:22:42.888355 (+ 173us) client_negotiation.cc:240] Negotiation successful
0924 03:22:42.888825 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":304,"threads_started":1}
I20260924 03:22:42.891665 20096 heartbeater.cc:347] Connected to a master server at 127.10.12.190:45917
I20260924 03:22:42.892058 20096 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:42.892874 20096 heartbeater.cc:511] Master 127.10.12.190:45917 requested a full tablet report, sending...
I20260924 03:22:42.894356 19649 ts_manager.cc:194] Registered new tserver with Master: cc7516968e6245bab986f2ef05422ecb (127.10.12.131:43767)
I20260924 03:22:42.895177 19649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.12.131:49995
I20260924 03:22:42.905810 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 03:22:42.912432 19837 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:42.910714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58928 (local address 127.10.12.190:45917)
0924 03:22:42.910954 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:42.910961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:42.911311 (+ 350us) server_negotiation.cc:407] Connection header received
0924 03:22:42.911497 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:42.911502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:42.911579 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:42.911667 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:42.912216 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58928: BlockingRecv error: recv got EOF from 127.0.0.1:58928 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 03:22:42.913201 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 19708
I20260924 03:22:42.921730 20105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:42.914136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33820 (local address 127.10.12.129:35743)
0924 03:22:42.914662 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:42.914669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:42.914696 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:22:42.914760 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:42.914766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:42.914992 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:42.915127 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:42.915576 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:42.916607 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:42.917947 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:42.918554 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:42.918666 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:42.919989 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:42.920002 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:42.920021 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:22:42.920648 (+ 627us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:42.920700 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:22:42.920728 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:42.920830 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:42.921136 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 03:22:42.921318 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":251,"threads_started":1}
I20260924 03:22:42.925282 19830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:43.018188 19835 heartbeater.cc:503] Master 127.10.12.190:45917 was elected leader, sending a full tablet report...
I20260924 03:22:43.019361 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 19708
W20260924 03:22:43.032665 19636 connection.cc:535] server connection from 127.10.12.129:41501 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 03:22:43.032853 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 19839
I20260924 03:22:43.041195 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:43.033824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32956 (local address 127.10.12.130:45077)
0924 03:22:43.034320 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:43.034328 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:43.034356 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:22:43.034447 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:43.034454 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:43.034669 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:43.034803 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:43.035291 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:43.036412 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:43.037728 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:43.038340 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:43.038461 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:43.039545 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:43.039553 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:43.039568 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:22:43.040227 (+ 659us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:43.040277 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:22:43.040344 (+ 67us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:43.040440 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:43.040645 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 03:22:43.040876 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":254,"threads_started":1}
I20260924 03:22:43.042495 19961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:43.125701 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 19839
I20260924 03:22:43.139731 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 19969
I20260924 03:22:43.149248 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:43.140747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55742 (local address 127.10.12.131:43767)
0924 03:22:43.141452 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:43.141462 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:43.141496 (+ 34us) server_negotiation.cc:407] Connection header received
0924 03:22:43.141560 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:43.141567 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:43.141811 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:43.141976 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:43.142449 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:43.143541 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:43.145010 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:43.145616 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:43.145744 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:43.147270 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:43.147286 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:43.147306 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:22:43.148267 (+ 961us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:43.148346 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:22:43.148379 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:43.148514 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:43.148713 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 03:22:43.148888 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":386,"threads_started":1}
I20260924 03:22:43.150700 20091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:43.233462 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 19969
I20260924 03:22:43.246145 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 19618
I20260924 03:22:43.247407 19679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:43.351275 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 19618
2026-09-24T03:22:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2530 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T03:22:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:22:43Z Disabled control of system clock
I20260924 03:22:43.420307 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:34597
--webserver_interface=127.10.12.190
--webserver_port=0
--builtin_ntp_servers=127.10.12.148:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:34597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 03:22:43.760366 20115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:43.760691 20115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:43.760783 20115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:43.768291 20115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:22:43.768427 20115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:43.768464 20115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:43.768528 20115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:22:43.768573 20115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:22:43.778090 20115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:34597
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:34597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:43.779398 20115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:43.781286 20115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:43.793294 20120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:43.793370 20121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:43.793399 20123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:43.793764 20115 server_base.cc:1034] running on GCE node
I20260924 03:22:43.794905 20115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:43.796463 20115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:43.797714 20115 hybrid_clock.cc:648] HybridClock initialized: now 1790220163797647 us; error 56 us; skew 500 ppm
I20260924 03:22:43.798134 20115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:22:43.800894 20115 webserver.cc:460] Webserver started at http://127.10.12.190:38417/ using document root <none> and password file <none>
I20260924 03:22:43.801679 20115 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:43.801797 20115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:43.802160 20115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:43.805383 20115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "e5b5382d9f76435eac02c061559989ee"
format_stamp: "Formatted at 2026-09-24 03:22:43 on dist-test-slave-00qs"
server_key: "c7c0caeefef44697fc16fe5b260e71bd"
server_key_iv: "f4225a7c89b15ecd10361e3b81166494"
server_key_version: "encryptionkey@0"
I20260924 03:22:43.806298 20115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "e5b5382d9f76435eac02c061559989ee"
format_stamp: "Formatted at 2026-09-24 03:22:43 on dist-test-slave-00qs"
server_key: "c7c0caeefef44697fc16fe5b260e71bd"
server_key_iv: "f4225a7c89b15ecd10361e3b81166494"
server_key_version: "encryptionkey@0"
I20260924 03:22:43.812139 20115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:22:43.816109 20129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:43.817761 20115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:22:43.818084 20115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "e5b5382d9f76435eac02c061559989ee"
format_stamp: "Formatted at 2026-09-24 03:22:43 on dist-test-slave-00qs"
server_key: "c7c0caeefef44697fc16fe5b260e71bd"
server_key_iv: "f4225a7c89b15ecd10361e3b81166494"
server_key_version: "encryptionkey@0"
I20260924 03:22:43.818324 20115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:43.824685 20115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:43.825529 20115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:43.825815 20115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:43.843879 20115 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:34597
I20260924 03:22:43.843899 20180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:34597 every 8 connection(s)
I20260924 03:22:43.845777 20115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:22:43.850497 20181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:43.855643 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 20115
I20260924 03:22:43.855856 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:22:43.856519 10290 external_mini_cluster.cc:1428] Setting key edeae0c4d4de6cbdd63cd4710c245b97
I20260924 03:22:43.862926 20181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5b5382d9f76435eac02c061559989ee: Bootstrap starting.
I20260924 03:22:43.867053 20181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5b5382d9f76435eac02c061559989ee: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:43.867573 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:43.859096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43324 (local address 127.10.12.190:34597)
0924 03:22:43.860363 (+ 1267us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:43.860380 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:43.860437 (+ 57us) server_negotiation.cc:407] Connection header received
0924 03:22:43.862733 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:43.862765 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:43.863371 (+ 606us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:43.863848 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:43.865403 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:43.865412 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:43.865430 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:22:43.865451 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:43.865541 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:22:43.865563 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:43.865794 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:43.865917 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 03:22:43.866216 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":369,"threads_started":1}
I20260924 03:22:43.868415 20181 log.cc:824] T 00000000000000000000000000000000 P e5b5382d9f76435eac02c061559989ee: Log is configured to *not* fsync() on all Append() calls
I20260924 03:22:43.871407 20181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5b5382d9f76435eac02c061559989ee: No bootstrap required, opened a new log
I20260924 03:22:43.879158 20181 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5b5382d9f76435eac02c061559989ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b5382d9f76435eac02c061559989ee" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 34597 } }
I20260924 03:22:43.879596 20181 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5b5382d9f76435eac02c061559989ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:43.879693 20181 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5b5382d9f76435eac02c061559989ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5b5382d9f76435eac02c061559989ee, State: Initialized, Role: FOLLOWER
I20260924 03:22:43.880352 20181 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5b5382d9f76435eac02c061559989ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b5382d9f76435eac02c061559989ee" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 34597 } }
I20260924 03:22:43.880599 20181 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5b5382d9f76435eac02c061559989ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:22:43.880730 20181 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5b5382d9f76435eac02c061559989ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:22:43.880895 20181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5b5382d9f76435eac02c061559989ee [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:43.883140 20181 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5b5382d9f76435eac02c061559989ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b5382d9f76435eac02c061559989ee" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 34597 } }
I20260924 03:22:43.883718 20181 leader_election.cc:302] T 00000000000000000000000000000000 P e5b5382d9f76435eac02c061559989ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5b5382d9f76435eac02c061559989ee; no voters:
I20260924 03:22:43.884443 20181 leader_election.cc:288] T 00000000000000000000000000000000 P e5b5382d9f76435eac02c061559989ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:22:43.884582 20186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5b5382d9f76435eac02c061559989ee [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:22:43.885269 20186 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5b5382d9f76435eac02c061559989ee [term 1 LEADER]: Becoming Leader. State: Replica: e5b5382d9f76435eac02c061559989ee, State: Running, Role: LEADER
I20260924 03:22:43.885836 20186 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5b5382d9f76435eac02c061559989ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b5382d9f76435eac02c061559989ee" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 34597 } }
I20260924 03:22:43.886739 20181 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5b5382d9f76435eac02c061559989ee [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:22:43.888760 20188 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5b5382d9f76435eac02c061559989ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5b5382d9f76435eac02c061559989ee. Latest consensus state: current_term: 1 leader_uuid: "e5b5382d9f76435eac02c061559989ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b5382d9f76435eac02c061559989ee" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 34597 } } }
I20260924 03:22:43.889170 20188 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5b5382d9f76435eac02c061559989ee [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:43.892201 20187 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5b5382d9f76435eac02c061559989ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5b5382d9f76435eac02c061559989ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b5382d9f76435eac02c061559989ee" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 34597 } } }
I20260924 03:22:43.892549 20187 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5b5382d9f76435eac02c061559989ee [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:43.893249 20194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:22:43.899085 20194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:22:43.909783 20194 catalog_manager.cc:1347] Generated new cluster ID: 22e23e4eb57f44a98202fa37526eadb8
I20260924 03:22:43.909973 20194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:22:43.942337 20194 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:22:43.942817 20194 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:22:43.950538 20194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5b5382d9f76435eac02c061559989ee: Generated new TSK 0
I20260924 03:22:43.951849 20194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:22:43.970465 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:34597
--builtin_ntp_servers=127.10.12.148:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 03:22:44.321568 20205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:44.322054 20205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:44.322212 20205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:44.330142 20205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:44.330271 20205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:44.330387 20205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:22:44.340231 20205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:34597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:44.341755 20205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:44.343571 20205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:44.357101 20211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:44.357134 20213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:44.357040 20210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:44.357945 20205 server_base.cc:1034] running on GCE node
I20260924 03:22:44.358590 20205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:44.359727 20205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:44.360960 20205 hybrid_clock.cc:648] HybridClock initialized: now 1790220164360895 us; error 60 us; skew 500 ppm
I20260924 03:22:44.361336 20205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:22:44.363746 20205 webserver.cc:460] Webserver started at http://127.10.12.129:41269/ using document root <none> and password file <none>
I20260924 03:22:44.364516 20205 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:44.364634 20205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:44.364948 20205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:44.368041 20205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "4aac9bdcda034faa8794d4eedfe9941d"
format_stamp: "Formatted at 2026-09-24 03:22:44 on dist-test-slave-00qs"
server_key: "2b256790cf2537312a2be6ee051148d8"
server_key_iv: "4ad33f7aa2bf715a2900ffad69580be1"
server_key_version: "encryptionkey@0"
I20260924 03:22:44.368902 20205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "4aac9bdcda034faa8794d4eedfe9941d"
format_stamp: "Formatted at 2026-09-24 03:22:44 on dist-test-slave-00qs"
server_key: "2b256790cf2537312a2be6ee051148d8"
server_key_iv: "4ad33f7aa2bf715a2900ffad69580be1"
server_key_version: "encryptionkey@0"
I20260924 03:22:44.374706 20205 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:22:44.378750 20219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:44.380662 20205 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260924 03:22:44.380864 20205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "4aac9bdcda034faa8794d4eedfe9941d"
format_stamp: "Formatted at 2026-09-24 03:22:44 on dist-test-slave-00qs"
server_key: "2b256790cf2537312a2be6ee051148d8"
server_key_iv: "4ad33f7aa2bf715a2900ffad69580be1"
server_key_version: "encryptionkey@0"
I20260924 03:22:44.381105 20205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:44.387264 20205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:44.388298 20205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:44.388583 20205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:44.389564 20205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:22:44.391094 20205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:22:44.391197 20205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:44.391315 20205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:22:44.391386 20205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:44.425918 20205 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:37369
I20260924 03:22:44.426010 20331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:37369 every 8 connection(s)
I20260924 03:22:44.427345 20205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:22:44.429236 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 20205
I20260924 03:22:44.429463 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:22:44.430095 10290 external_mini_cluster.cc:1428] Setting key 010f4dbae50f1d1b0001ccc42f3b62f2
I20260924 03:22:44.433154 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:34597
--builtin_ntp_servers=127.10.12.148:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 03:22:44.437800 20334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:44.430664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:52489 (local address 127.10.12.190:34597)
0924 03:22:44.431186 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:44.431191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:44.432816 (+ 1625us) server_negotiation.cc:407] Connection header received
0924 03:22:44.434807 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:44.434813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:44.434898 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:44.435032 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:44.436960 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:44.436967 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:44.436971 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:22:44.436998 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:44.437067 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:22:44.437083 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:44.437217 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:44.437412 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 03:22:44.437481 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":300,"threads_started":1}
I20260924 03:22:44.438894 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:44.431401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:34597 (local address 127.10.12.129:52489)
0924 03:22:44.432611 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:44.432672 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:44.434389 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:44.435268 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:44.435277 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:44.435390 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:44.436379 (+ 989us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:22:44.436383 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:44.436454 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:22:44.436616 (+ 162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:44.437202 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:44.437213 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:44.437220 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 03:22:44.437477 (+ 257us) client_negotiation.cc:240] Negotiation successful
0924 03:22:44.437757 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":341,"threads_started":1}
I20260924 03:22:44.440790 20332 heartbeater.cc:347] Connected to a master server at 127.10.12.190:34597
I20260924 03:22:44.441152 20332 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:44.441814 20332 heartbeater.cc:511] Master 127.10.12.190:34597 requested a full tablet report, sending...
I20260924 03:22:44.444718 20146 ts_manager.cc:194] Registered new tserver with Master: 4aac9bdcda034faa8794d4eedfe9941d (127.10.12.129:37369)
W20260924 03:22:44.781018 20336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:44.781319 20336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:44.781630 20336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:44.789616 20336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:44.789750 20336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:44.789882 20336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:22:44.799672 20336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:34597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:44.801048 20336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:44.802886 20336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:44.816550 20341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:44.816687 20342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:44.816758 20344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:44.817822 20336 server_base.cc:1034] running on GCE node
I20260924 03:22:44.818509 20336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:44.819583 20336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:44.820797 20336 hybrid_clock.cc:648] HybridClock initialized: now 1790220164820732 us; error 55 us; skew 500 ppm
I20260924 03:22:44.821130 20336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:22:44.823577 20336 webserver.cc:460] Webserver started at http://127.10.12.130:42683/ using document root <none> and password file <none>
I20260924 03:22:44.824523 20336 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:44.824638 20336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:44.824951 20336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:44.827903 20336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "3cc5eafc4702449a8522a7c41c0c2b92"
format_stamp: "Formatted at 2026-09-24 03:22:44 on dist-test-slave-00qs"
server_key: "90266fddd5ce8cc35b088b2c28971c58"
server_key_iv: "05b40fb96c0f3aeab0714ae2a6d11d2e"
server_key_version: "encryptionkey@0"
I20260924 03:22:44.828785 20336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "3cc5eafc4702449a8522a7c41c0c2b92"
format_stamp: "Formatted at 2026-09-24 03:22:44 on dist-test-slave-00qs"
server_key: "90266fddd5ce8cc35b088b2c28971c58"
server_key_iv: "05b40fb96c0f3aeab0714ae2a6d11d2e"
server_key_version: "encryptionkey@0"
I20260924 03:22:44.834259 20336 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:22:44.838217 20350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:44.839840 20336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:22:44.840301 20336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "3cc5eafc4702449a8522a7c41c0c2b92"
format_stamp: "Formatted at 2026-09-24 03:22:44 on dist-test-slave-00qs"
server_key: "90266fddd5ce8cc35b088b2c28971c58"
server_key_iv: "05b40fb96c0f3aeab0714ae2a6d11d2e"
server_key_version: "encryptionkey@0"
I20260924 03:22:44.840705 20336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:44.848429 20336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:44.849503 20336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:44.849799 20336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:44.850934 20336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:22:44.852856 20336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:22:44.852967 20336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:44.853070 20336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:22:44.853135 20336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:44.889428 20336 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:40657
I20260924 03:22:44.889479 20462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:40657 every 8 connection(s)
I20260924 03:22:44.890765 20336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:22:44.900583 20334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:44.894005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:39193 (local address 127.10.12.190:34597)
0924 03:22:44.894433 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:44.894440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:44.896076 (+ 1636us) server_negotiation.cc:407] Connection header received
0924 03:22:44.897880 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:44.897886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:44.897951 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:44.898052 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:44.899518 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:44.899523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:44.899526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:22:44.899546 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:44.899580 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:22:44.899593 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:44.899710 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:44.900114 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 03:22:44.900283 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260924 03:22:44.901119 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 20336
I20260924 03:22:44.901322 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:22:44.901510 20465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:44.894674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:34597 (local address 127.10.12.130:39193)
0924 03:22:44.895796 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:44.895841 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:44.897527 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:44.898347 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:44.898358 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:44.898466 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:44.899149 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:22:44.899152 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:44.899194 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:22:44.899289 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:44.899689 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:44.899697 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:44.899704 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 03:22:44.899944 (+ 240us) client_negotiation.cc:240] Negotiation successful
0924 03:22:44.900227 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":368,"threads_started":1}
I20260924 03:22:44.901939 10290 external_mini_cluster.cc:1428] Setting key ba0c45f7ffe4a6e97122a10602bd3672
I20260924 03:22:44.903173 20463 heartbeater.cc:347] Connected to a master server at 127.10.12.190:34597
I20260924 03:22:44.903481 20463 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:44.904151 20463 heartbeater.cc:511] Master 127.10.12.190:34597 requested a full tablet report, sending...
I20260924 03:22:44.904803 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:34597
--builtin_ntp_servers=127.10.12.148:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 03:22:44.905756 20146 ts_manager.cc:194] Registered new tserver with Master: 3cc5eafc4702449a8522a7c41c0c2b92 (127.10.12.130:40657)
W20260924 03:22:45.242108 20466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:45.242405 20466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:45.242494 20466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:45.249971 20466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:45.250093 20466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:45.250206 20466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:22:45.260134 20466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:34597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:45.261489 20466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:45.263362 20466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:45.277199 20471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:45.277128 20472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:45.277159 20474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:45.278216 20466 server_base.cc:1034] running on GCE node
I20260924 03:22:45.278985 20466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:45.280128 20466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:45.281386 20466 hybrid_clock.cc:648] HybridClock initialized: now 1790220165281345 us; error 40 us; skew 500 ppm
I20260924 03:22:45.281747 20466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:22:45.284379 20466 webserver.cc:460] Webserver started at http://127.10.12.131:38099/ using document root <none> and password file <none>
I20260924 03:22:45.285256 20466 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:45.285373 20466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:45.285800 20466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:45.289392 20466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "a76f910b4fab4e9b92e2374a53e69d50"
format_stamp: "Formatted at 2026-09-24 03:22:45 on dist-test-slave-00qs"
server_key: "61082852ebfec97030ed7eceb51bfd84"
server_key_iv: "98d99adeac69f1f47e6b75e7154bb8a6"
server_key_version: "encryptionkey@0"
I20260924 03:22:45.290234 20466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "a76f910b4fab4e9b92e2374a53e69d50"
format_stamp: "Formatted at 2026-09-24 03:22:45 on dist-test-slave-00qs"
server_key: "61082852ebfec97030ed7eceb51bfd84"
server_key_iv: "98d99adeac69f1f47e6b75e7154bb8a6"
server_key_version: "encryptionkey@0"
I20260924 03:22:45.296129 20466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:22:45.300161 20480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:45.301668 20466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:22:45.301854 20466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "a76f910b4fab4e9b92e2374a53e69d50"
format_stamp: "Formatted at 2026-09-24 03:22:45 on dist-test-slave-00qs"
server_key: "61082852ebfec97030ed7eceb51bfd84"
server_key_iv: "98d99adeac69f1f47e6b75e7154bb8a6"
server_key_version: "encryptionkey@0"
I20260924 03:22:45.302109 20466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:45.308478 20466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:45.309392 20466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:45.309680 20466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:45.310649 20466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:22:45.312335 20466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:22:45.312464 20466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:45.312577 20466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:22:45.312657 20466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:45.344902 20466 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:44923
I20260924 03:22:45.344970 20592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:44923 every 8 connection(s)
I20260924 03:22:45.346308 20466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:22:45.348557 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 20466
I20260924 03:22:45.348757 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:22:45.349365 10290 external_mini_cluster.cc:1428] Setting key 4b220278c1d4e35a1ac754e49f31d7ae
I20260924 03:22:45.355531 20334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:45.349439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:36925 (local address 127.10.12.190:34597)
0924 03:22:45.349625 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:45.349630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:45.351346 (+ 1716us) server_negotiation.cc:407] Connection header received
0924 03:22:45.353288 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:45.353293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:45.353372 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:45.353461 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:45.354802 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:45.354809 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:45.354814 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:22:45.354833 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:45.354865 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:22:45.354877 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:45.354994 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:45.355175 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 03:22:45.355248 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 03:22:45.356604 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:45.350088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:34597 (local address 127.10.12.131:36925)
0924 03:22:45.351198 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:45.351237 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:45.353016 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:45.353711 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:45.353721 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:45.353826 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:45.354489 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:22:45.354491 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:45.354531 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:22:45.354628 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:45.354970 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:45.354981 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:45.354989 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 03:22:45.355258 (+ 269us) client_negotiation.cc:240] Negotiation successful
0924 03:22:45.355573 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":300,"threads_started":1}
I20260924 03:22:45.357935 20593 heartbeater.cc:347] Connected to a master server at 127.10.12.190:34597
I20260924 03:22:45.358218 20593 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:45.358788 20593 heartbeater.cc:511] Master 127.10.12.190:34597 requested a full tablet report, sending...
I20260924 03:22:45.360571 20146 ts_manager.cc:194] Registered new tserver with Master: a76f910b4fab4e9b92e2374a53e69d50 (127.10.12.131:44923)
I20260924 03:22:45.365425 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 03:22:45.371860 20334 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:22:45.370270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43334 (local address 127.10.12.190:34597)
0924 03:22:45.370436 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:45.370452 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:45.370886 (+ 434us) server_negotiation.cc:407] Connection header received
0924 03:22:45.371063 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:45.371067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:45.371122 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:45.371202 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:45.371587 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43334: BlockingRecv error: recv got EOF from 127.0.0.1:43334 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 03:22:45.372469 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 20205
I20260924 03:22:45.376180 20602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:45.373658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45446 (local address 127.10.12.129:37369)
0924 03:22:45.374147 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:45.374154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:45.374181 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:22:45.374247 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:45.374254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:45.374496 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:45.374637 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:45.375544 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:45.375551 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:45.375563 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:22:45.375583 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:45.375625 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:22:45.375640 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:45.375744 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:45.375798 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 03:22:45.375878 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":236,"threads_started":1}
I20260924 03:22:45.377461 20327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:45.458946 20332 heartbeater.cc:503] Master 127.10.12.190:34597 was elected leader, sending a full tablet report...
I20260924 03:22:45.460173 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 20205
W20260924 03:22:45.474352 20131 connection.cc:535] server connection from 127.10.12.129:52489 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 03:22:45.474434 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 20336
I20260924 03:22:45.478039 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:45.475315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43390 (local address 127.10.12.130:40657)
0924 03:22:45.475825 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:45.475833 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:45.475870 (+ 37us) server_negotiation.cc:407] Connection header received
0924 03:22:45.475928 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:45.475935 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:45.476213 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:45.476361 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:45.477321 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:45.477327 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:45.477338 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:22:45.477356 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:45.477399 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:22:45.477413 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:45.477535 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:45.477664 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 03:22:45.477768 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":251,"threads_started":1}
I20260924 03:22:45.479301 20458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:45.561120 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 20336
I20260924 03:22:45.574075 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 20466
I20260924 03:22:45.578161 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:45.575016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51590 (local address 127.10.12.131:44923)
0924 03:22:45.575629 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:45.575636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:45.575666 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:22:45.575728 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:45.575734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:45.576041 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:45.576206 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:45.577325 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:45.577331 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:45.577343 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:22:45.577359 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:45.577398 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:22:45.577411 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:45.577578 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:45.577651 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 03:22:45.577753 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":296,"threads_started":1}
I20260924 03:22:45.579595 20588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:45.662030 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 20466
I20260924 03:22:45.674250 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 20115
I20260924 03:22:45.675493 20176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:45.778532 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 20115
2026-09-24T03:22:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2426 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:22:45 dist-test-slave-00qs krb5kdc[20611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:22:45 dist-test-slave-00qs krb5kdc[20611](info): set up 2 sockets
Sep 24 03:22:45 dist-test-slave-00qs krb5kdc[20611](info): commencing operation
krb5kdc: starting...
W20260924 03:22:46.825178 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:22:46 dist-test-slave-00qs krb5kdc[20611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:22:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:22:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:22:47.065555 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:41373
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:39947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:41373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:22:47.410475 20624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:47.410820 20624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:47.410943 20624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:47.419548 20624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:22:47.419986 20624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:47.420152 20624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:47.420257 20624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:22:47.420337 20624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:22:47.431495 20624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:39947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:41373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:41373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:47.432960 20624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:47.434279 20624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:47.446539 20630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:47.446666 20629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:47.446652 20632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:47.448100 20624 server_base.cc:1034] running on GCE node
I20260924 03:22:47.448973 20624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:47.450533 20624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:47.451772 20624 hybrid_clock.cc:648] HybridClock initialized: now 1790220167451716 us; error 45 us; skew 500 ppm
I20260924 03:22:47.452227 20624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:47 dist-test-slave-00qs krb5kdc[20611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220167, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:47.460383 20624 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:22:47.462316 20624 webserver.cc:460] Webserver started at http://127.10.12.190:43611/ using document root <none> and password file <none>
I20260924 03:22:47.463088 20624 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:47.463212 20624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:47.463490 20624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:47.466640 20624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "c54476d20b804d4093ae6dc394b72a8a"
format_stamp: "Formatted at 2026-09-24 03:22:47 on dist-test-slave-00qs"
server_key: "51ac1d486026257bc22733cdc3f9760f"
server_key_iv: "956338073ac07b457f381061d4619d94"
server_key_version: "encryptionkey@0"
I20260924 03:22:47.467442 20624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "c54476d20b804d4093ae6dc394b72a8a"
format_stamp: "Formatted at 2026-09-24 03:22:47 on dist-test-slave-00qs"
server_key: "51ac1d486026257bc22733cdc3f9760f"
server_key_iv: "956338073ac07b457f381061d4619d94"
server_key_version: "encryptionkey@0"
I20260924 03:22:47.473205 20624 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 03:22:47.476965 20639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:47.478637 20624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:22:47.478818 20624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "c54476d20b804d4093ae6dc394b72a8a"
format_stamp: "Formatted at 2026-09-24 03:22:47 on dist-test-slave-00qs"
server_key: "51ac1d486026257bc22733cdc3f9760f"
server_key_iv: "956338073ac07b457f381061d4619d94"
server_key_version: "encryptionkey@0"
I20260924 03:22:47.479017 20624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:47.503789 20624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:47.508451 20624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:47.508718 20624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:47.528198 20624 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:41373
I20260924 03:22:47.528209 20690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:41373 every 8 connection(s)
I20260924 03:22:47.529641 20624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:22:47.530772 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 20624
I20260924 03:22:47.531016 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:22:47.531589 10290 external_mini_cluster.cc:1428] Setting key 7b8637624a0c0f51e80d19e7e9d35c25
I20260924 03:22:47.534500 20691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 03:22:47 dist-test-slave-00qs krb5kdc[20611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:22:47.545706 20691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c54476d20b804d4093ae6dc394b72a8a: Bootstrap starting.
I20260924 03:22:47.549070 20691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c54476d20b804d4093ae6dc394b72a8a: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:47.550323 20691 log.cc:824] T 00000000000000000000000000000000 P c54476d20b804d4093ae6dc394b72a8a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:22:47.553406 20691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c54476d20b804d4093ae6dc394b72a8a: No bootstrap required, opened a new log
I20260924 03:22:47.554014 20694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:47.533678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45286 (local address 127.10.12.190:41373)
0924 03:22:47.534503 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:47.534516 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:47.534559 (+ 43us) server_negotiation.cc:407] Connection header received
0924 03:22:47.536323 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:47.536347 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:47.536760 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:47.537178 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:47.538688 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:47.540096 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:47.541388 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:47.542114 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:47.542298 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:47.546438 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:47.546446 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:47.546462 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:47.546523 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:47.549548 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:47.550477 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:47.550486 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:47.550495 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:47.550676 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:47.551137 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:47.551142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:47.551150 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:47.551726 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:47.552057 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:47.552513 (+ 456us) server_negotiation.cc:299] Negotiation successful
0924 03:22:47.552978 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":363,"threads_started":1}
I20260924 03:22:47.561324 20691 raft_consensus.cc:348] T 00000000000000000000000000000000 P c54476d20b804d4093ae6dc394b72a8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54476d20b804d4093ae6dc394b72a8a" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 41373 } }
I20260924 03:22:47.561743 20691 raft_consensus.cc:374] T 00000000000000000000000000000000 P c54476d20b804d4093ae6dc394b72a8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:47.561836 20691 raft_consensus.cc:729] T 00000000000000000000000000000000 P c54476d20b804d4093ae6dc394b72a8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c54476d20b804d4093ae6dc394b72a8a, State: Initialized, Role: FOLLOWER
I20260924 03:22:47.562479 20691 consensus_queue.cc:260] T 00000000000000000000000000000000 P c54476d20b804d4093ae6dc394b72a8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54476d20b804d4093ae6dc394b72a8a" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 41373 } }
I20260924 03:22:47.562793 20691 raft_consensus.cc:388] T 00000000000000000000000000000000 P c54476d20b804d4093ae6dc394b72a8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:22:47.562934 20691 raft_consensus.cc:482] T 00000000000000000000000000000000 P c54476d20b804d4093ae6dc394b72a8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:22:47.563102 20691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c54476d20b804d4093ae6dc394b72a8a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:47.565491 20691 raft_consensus.cc:504] T 00000000000000000000000000000000 P c54476d20b804d4093ae6dc394b72a8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54476d20b804d4093ae6dc394b72a8a" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 41373 } }
I20260924 03:22:47.566042 20691 leader_election.cc:302] T 00000000000000000000000000000000 P c54476d20b804d4093ae6dc394b72a8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c54476d20b804d4093ae6dc394b72a8a; no voters:
I20260924 03:22:47.566630 20691 leader_election.cc:288] T 00000000000000000000000000000000 P c54476d20b804d4093ae6dc394b72a8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:22:47.566779 20696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c54476d20b804d4093ae6dc394b72a8a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:22:47.567451 20696 raft_consensus.cc:686] T 00000000000000000000000000000000 P c54476d20b804d4093ae6dc394b72a8a [term 1 LEADER]: Becoming Leader. State: Replica: c54476d20b804d4093ae6dc394b72a8a, State: Running, Role: LEADER
I20260924 03:22:47.568054 20696 consensus_queue.cc:237] T 00000000000000000000000000000000 P c54476d20b804d4093ae6dc394b72a8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54476d20b804d4093ae6dc394b72a8a" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 41373 } }
I20260924 03:22:47.568748 20691 sys_catalog.cc:564] T 00000000000000000000000000000000 P c54476d20b804d4093ae6dc394b72a8a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:22:47.571008 20697 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54476d20b804d4093ae6dc394b72a8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c54476d20b804d4093ae6dc394b72a8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54476d20b804d4093ae6dc394b72a8a" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 41373 } } }
I20260924 03:22:47.571096 20698 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54476d20b804d4093ae6dc394b72a8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c54476d20b804d4093ae6dc394b72a8a. Latest consensus state: current_term: 1 leader_uuid: "c54476d20b804d4093ae6dc394b72a8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54476d20b804d4093ae6dc394b72a8a" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 41373 } } }
I20260924 03:22:47.571599 20698 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54476d20b804d4093ae6dc394b72a8a [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:47.571653 20697 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54476d20b804d4093ae6dc394b72a8a [sys.catalog]: This master's current role is: LEADER
I20260924 03:22:47.572789 20703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:22:47.578962 20703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:22:47.587996 20703 catalog_manager.cc:1347] Generated new cluster ID: 4ac165bd8bf34230b782c27052fe3101
I20260924 03:22:47.588127 20703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:22:47.604390 20703 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:22:47.605760 20703 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:22:47.617920 20703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c54476d20b804d4093ae6dc394b72a8a: Generated new TSK 0
I20260924 03:22:47.618810 20703 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
I20260924 03:22:47.677568 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:41373
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:39947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:22:48.048913 20717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:48.049258 20717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:48.049365 20717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:48.056986 20717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:48.057116 20717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:48.057349 20717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:22:48.067914 20717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:39947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:41373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:48.069372 20717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:48.070724 20717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:48.086299 20723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:48.086223 20722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:48.086256 20725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:48.086915 20717 server_base.cc:1034] running on GCE node
I20260924 03:22:48.087621 20717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:48.089172 20717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:48.090540 20717 hybrid_clock.cc:648] HybridClock initialized: now 1790220168090451 us; error 86 us; skew 500 ppm
I20260924 03:22:48.091037 20717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:48 dist-test-slave-00qs krb5kdc[20611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220168, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:48.094748 20717 init.cc:377] Logged in from keytab as kudu/127.10.12.129@KRBTEST.COM (short username kudu)
I20260924 03:22:48.096393 20717 webserver.cc:460] Webserver started at http://127.10.12.129:43797/ using document root <none> and password file <none>
I20260924 03:22:48.097110 20717 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:48.097224 20717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:48.097517 20717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:48.100567 20717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "972d95236f9b481aa10d3cb7c460c854"
format_stamp: "Formatted at 2026-09-24 03:22:48 on dist-test-slave-00qs"
server_key: "1f571e9530cfef95228d9a42c9730d3f"
server_key_iv: "64e65412d48dd6c2bea89fea61975623"
server_key_version: "encryptionkey@0"
I20260924 03:22:48.101414 20717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "972d95236f9b481aa10d3cb7c460c854"
format_stamp: "Formatted at 2026-09-24 03:22:48 on dist-test-slave-00qs"
server_key: "1f571e9530cfef95228d9a42c9730d3f"
server_key_iv: "64e65412d48dd6c2bea89fea61975623"
server_key_version: "encryptionkey@0"
I20260924 03:22:48.106953 20717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:22:48.111061 20732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:48.112799 20717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:22:48.113003 20717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "972d95236f9b481aa10d3cb7c460c854"
format_stamp: "Formatted at 2026-09-24 03:22:48 on dist-test-slave-00qs"
server_key: "1f571e9530cfef95228d9a42c9730d3f"
server_key_iv: "64e65412d48dd6c2bea89fea61975623"
server_key_version: "encryptionkey@0"
I20260924 03:22:48.113212 20717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:48.137817 20717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:48.142331 20717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:48.142642 20717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:48.143651 20717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:22:48.145304 20717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:22:48.145391 20717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:48.145520 20717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:22:48.145574 20717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:48.178619 20717 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:42053
I20260924 03:22:48.178665 20844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:42053 every 8 connection(s)
I20260924 03:22:48.179888 20717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:22:48.188764 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 20717
I20260924 03:22:48.189002 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:22:48.189626 10290 external_mini_cluster.cc:1428] Setting key 357d34bf1ae5c5bf08a7b068e3592715
Sep 24 03:22:48 dist-test-slave-00qs krb5kdc[20611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220168, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:22:48.202905 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:48.182625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:59799 (local address 127.10.12.190:41373)
0924 03:22:48.183085 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:48.183090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:48.184103 (+ 1013us) server_negotiation.cc:407] Connection header received
0924 03:22:48.186175 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:48.186180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:48.186271 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:48.186366 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:48.188062 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:48.189037 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:48.190979 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:48.191571 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:48.191696 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:48.196430 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:48.196445 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:48.196450 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:48.196493 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:48.199141 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:48.200115 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:48.200121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:48.200124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:48.200276 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:48.200778 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:48.200783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:48.200787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:48.201135 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:48.201284 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:48.202186 (+ 902us) server_negotiation.cc:299] Negotiation successful
0924 03:22:48.202574 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":221,"threads_started":1}
I20260924 03:22:48.203680 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:48.183179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:41373 (local address 127.10.12.129:59799)
0924 03:22:48.183901 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:48.183987 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:48.185841 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:48.186643 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:48.186653 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:48.187224 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:48.187849 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:48.187868 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:48.189246 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:48.189252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:48.190807 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:48.190824 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:48.191830 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:48.191839 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:48.192208 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:48.193022 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:48.193054 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:48.195885 (+ 2831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:48.199396 (+ 3511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:48.199405 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:48.199422 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:48.199940 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:48.200428 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:48.200432 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:48.200434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:48.200659 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:48.201326 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:48.201336 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:48.201736 (+ 400us) client_negotiation.cc:769] Sending connection context
0924 03:22:48.202035 (+ 299us) client_negotiation.cc:240] Negotiation successful
0924 03:22:48.202512 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":293,"threads_started":1}
I20260924 03:22:48.205408 20845 heartbeater.cc:347] Connected to a master server at 127.10.12.190:41373
I20260924 03:22:48.205780 20845 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:48.206668 20845 heartbeater.cc:511] Master 127.10.12.190:41373 requested a full tablet report, sending...
I20260924 03:22:48.209251 20656 ts_manager.cc:194] Registered new tserver with Master: 972d95236f9b481aa10d3cb7c460c854 (127.10.12.129:42053)
I20260924 03:22:48.211031 20656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.129@KRBTEST.COM'} at 127.10.12.129:59799
WARNING: no policy specified for kudu/127.10.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
I20260924 03:22:48.248234 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:41373
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:39947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:22:48.618194 20851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:48.618549 20851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:48.618691 20851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:48.626389 20851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:48.626545 20851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:48.626699 20851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:22:48.637272 20851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:39947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:41373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:48.638866 20851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:48.640316 20851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:48.654393 20859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:48.654422 20856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:48.654631 20857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:48.655635 20851 server_base.cc:1034] running on GCE node
I20260924 03:22:48.656548 20851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:48.657639 20851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:48.658895 20851 hybrid_clock.cc:648] HybridClock initialized: now 1790220168658836 us; error 59 us; skew 500 ppm
I20260924 03:22:48.659245 20851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:48 dist-test-slave-00qs krb5kdc[20611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220168, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:48.662508 20851 init.cc:377] Logged in from keytab as kudu/127.10.12.130@KRBTEST.COM (short username kudu)
I20260924 03:22:48.664268 20851 webserver.cc:460] Webserver started at http://127.10.12.130:33417/ using document root <none> and password file <none>
I20260924 03:22:48.664984 20851 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:48.665094 20851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:48.665355 20851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:48.668407 20851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "ce000ff670be4258892c0da863864d4a"
format_stamp: "Formatted at 2026-09-24 03:22:48 on dist-test-slave-00qs"
server_key: "332752c65a8af3f36c263b5ab466d5bb"
server_key_iv: "c4dde0b5968aa21ec96c71fbff16bce6"
server_key_version: "encryptionkey@0"
I20260924 03:22:48.669229 20851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "ce000ff670be4258892c0da863864d4a"
format_stamp: "Formatted at 2026-09-24 03:22:48 on dist-test-slave-00qs"
server_key: "332752c65a8af3f36c263b5ab466d5bb"
server_key_iv: "c4dde0b5968aa21ec96c71fbff16bce6"
server_key_version: "encryptionkey@0"
I20260924 03:22:48.674883 20851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260924 03:22:48.678715 20866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:48.680346 20851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:22:48.680511 20851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "ce000ff670be4258892c0da863864d4a"
format_stamp: "Formatted at 2026-09-24 03:22:48 on dist-test-slave-00qs"
server_key: "332752c65a8af3f36c263b5ab466d5bb"
server_key_iv: "c4dde0b5968aa21ec96c71fbff16bce6"
server_key_version: "encryptionkey@0"
I20260924 03:22:48.680704 20851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:48.700896 20851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:48.705514 20851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:48.705789 20851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:48.706916 20851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:22:48.708443 20851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:22:48.708523 20851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:48.708612 20851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:22:48.708657 20851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:48.741874 20851 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:35241
I20260924 03:22:48.741974 20978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:35241 every 8 connection(s)
I20260924 03:22:48.743144 20851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:22:48.753487 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 20851
I20260924 03:22:48.753693 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:22:48.754101 10290 external_mini_cluster.cc:1428] Setting key 190d78ec70a0d9d9460c11709e4cff91
Sep 24 03:22:48 dist-test-slave-00qs krb5kdc[20611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220168, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:22:48.764245 20981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:48.745723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:36869 (local address 127.10.12.190:41373)
0924 03:22:48.746182 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:48.746188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:48.747214 (+ 1026us) server_negotiation.cc:407] Connection header received
0924 03:22:48.749128 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:48.749132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:48.749208 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:48.749310 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:48.750973 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:48.751752 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:48.753479 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:48.754005 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:48.754126 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:48.758230 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:48.758235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:48.758238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:48.758275 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:48.760979 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:48.761829 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:48.761834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:48.761836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:48.761976 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:48.762420 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:48.762424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:48.762428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:48.762706 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:48.762853 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:48.763593 (+ 740us) server_negotiation.cc:299] Negotiation successful
0924 03:22:48.763839 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":208,"threads_started":1}
I20260924 03:22:48.765213 20982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:48.746307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:41373 (local address 127.10.12.130:36869)
0924 03:22:48.747048 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:48.747088 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:48.748783 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:48.749610 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:48.749619 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:48.750151 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:48.750749 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:48.750767 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:48.751941 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:48.751945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:48.753278 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:48.753292 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:48.754255 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:48.754263 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:48.754517 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:48.755332 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:48.755373 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:48.757956 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:48.761171 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:48.761182 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:48.761198 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:48.761686 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:48.762088 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:48.762092 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:48.762096 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:48.762301 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:48.762836 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:48.762845 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:48.763220 (+ 375us) client_negotiation.cc:769] Sending connection context
0924 03:22:48.763478 (+ 258us) client_negotiation.cc:240] Negotiation successful
0924 03:22:48.764080 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":286,"threads_started":1}
I20260924 03:22:48.766650 20979 heartbeater.cc:347] Connected to a master server at 127.10.12.190:41373
I20260924 03:22:48.766993 20979 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:48.767891 20979 heartbeater.cc:511] Master 127.10.12.190:41373 requested a full tablet report, sending...
I20260924 03:22:48.769711 20656 ts_manager.cc:194] Registered new tserver with Master: ce000ff670be4258892c0da863864d4a (127.10.12.130:35241)
I20260924 03:22:48.770601 20656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.130@KRBTEST.COM'} at 127.10.12.130:36869
WARNING: no policy specified for kudu/127.10.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
I20260924 03:22:48.808171 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:41373
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:39947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:22:49.149498 20985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:22:49.149829 20985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:22:49.149973 20985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:22:49.157760 20985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:22:49.157915 20985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:22:49.158083 20985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:22:49.168370 20985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:39947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:41373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:22:49.169750 20985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:22:49.171260 20985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:22:49.187498 20990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:49.187498 20991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:22:49.187732 20993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:22:49.188532 20985 server_base.cc:1034] running on GCE node
I20260924 03:22:49.189208 20985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:22:49.190304 20985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:22:49.191519 20985 hybrid_clock.cc:648] HybridClock initialized: now 1790220169191453 us; error 63 us; skew 500 ppm
I20260924 03:22:49.191852 20985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:22:49 dist-test-slave-00qs krb5kdc[20611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220169, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:22:49.195353 20985 init.cc:377] Logged in from keytab as kudu/127.10.12.131@KRBTEST.COM (short username kudu)
I20260924 03:22:49.197073 20985 webserver.cc:460] Webserver started at http://127.10.12.131:46867/ using document root <none> and password file <none>
I20260924 03:22:49.197831 20985 fs_manager.cc:359] Metadata directory not provided
I20260924 03:22:49.197974 20985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:22:49.198279 20985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:22:49.201490 20985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "2e6f98d5fc814c94b673ccb5e1a0d89e"
format_stamp: "Formatted at 2026-09-24 03:22:49 on dist-test-slave-00qs"
server_key: "0388d6407a07057c0c7adb50a5fcd619"
server_key_iv: "04184960beb9e54d73a57d140123469d"
server_key_version: "encryptionkey@0"
I20260924 03:22:49.202333 20985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "2e6f98d5fc814c94b673ccb5e1a0d89e"
format_stamp: "Formatted at 2026-09-24 03:22:49 on dist-test-slave-00qs"
server_key: "0388d6407a07057c0c7adb50a5fcd619"
server_key_iv: "04184960beb9e54d73a57d140123469d"
server_key_version: "encryptionkey@0"
I20260924 03:22:49.208333 20985 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 03:22:49.212460 21000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:49.214171 20985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:22:49.214375 20985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "2e6f98d5fc814c94b673ccb5e1a0d89e"
format_stamp: "Formatted at 2026-09-24 03:22:49 on dist-test-slave-00qs"
server_key: "0388d6407a07057c0c7adb50a5fcd619"
server_key_iv: "04184960beb9e54d73a57d140123469d"
server_key_version: "encryptionkey@0"
I20260924 03:22:49.214587 20985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:22:49.215142 20845 heartbeater.cc:503] Master 127.10.12.190:41373 was elected leader, sending a full tablet report...
I20260924 03:22:49.234045 20985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:22:49.238834 20985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:22:49.239145 20985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:22:49.240266 20985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:22:49.241765 20985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:22:49.241850 20985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:49.241978 20985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:22:49.242031 20985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:22:49.278620 20985 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:44667
I20260924 03:22:49.278743 21112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:44667 every 8 connection(s)
I20260924 03:22:49.280260 20985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:22:49.282966 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 20985
I20260924 03:22:49.283177 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:22:49.283701 10290 external_mini_cluster.cc:1428] Setting key 29a2fc6a502d2f562650f17a8fd6fc33
Sep 24 03:22:49 dist-test-slave-00qs krb5kdc[20611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220169, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:22:49.305104 21115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:49.283226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:56461 (local address 127.10.12.190:41373)
0924 03:22:49.283829 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:49.283836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:49.285069 (+ 1233us) server_negotiation.cc:407] Connection header received
0924 03:22:49.287495 (+ 2426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:49.287501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:49.287602 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:49.287741 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:49.289718 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.290544 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.292455 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.292887 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.293003 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:49.296929 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:49.296934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:49.296938 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:49.296980 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:49.300927 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:49.302184 (+ 1257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:49.302188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:49.302191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:49.302331 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:49.302869 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:49.302874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:49.302876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:49.303225 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:49.303354 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:49.304558 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0924 03:22:49.304793 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":283,"threads_started":1}
I20260924 03:22:49.306382 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:49.283980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:41373 (local address 127.10.12.131:56461)
0924 03:22:49.284913 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:49.284971 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:49.287124 (+ 2153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:49.288272 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:49.288282 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:49.288842 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:22:49.289475 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:49.289495 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.290742 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.290746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:49.292238 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:49.292255 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.293174 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.293180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:49.293424 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:49.294155 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:22:49.294186 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:22:49.296543 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:22:49.301379 (+ 4836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:49.301392 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:49.301412 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:49.302030 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:49.302474 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:22:49.302478 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:22:49.302481 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:22:49.302704 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:22:49.303381 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:22:49.303389 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:22:49.303810 (+ 421us) client_negotiation.cc:769] Sending connection context
0924 03:22:49.304185 (+ 375us) client_negotiation.cc:240] Negotiation successful
0924 03:22:49.305022 (+ 837us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":362,"threads_started":1}
I20260924 03:22:49.308871 21113 heartbeater.cc:347] Connected to a master server at 127.10.12.190:41373
I20260924 03:22:49.309211 21113 heartbeater.cc:464] Registering TS with master...
I20260924 03:22:49.310041 21113 heartbeater.cc:511] Master 127.10.12.190:41373 requested a full tablet report, sending...
I20260924 03:22:49.312049 20656 ts_manager.cc:194] Registered new tserver with Master: 2e6f98d5fc814c94b673ccb5e1a0d89e (127.10.12.131:44667)
I20260924 03:22:49.313478 20656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.131@KRBTEST.COM'} at 127.10.12.131:56461
I20260924 03:22:49.315763 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:22:49.336318 21115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:49.321694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45298 (local address 127.10.12.190:41373)
0924 03:22:49.321910 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:49.321917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:49.322278 (+ 361us) server_negotiation.cc:407] Connection header received
0924 03:22:49.322388 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:49.322392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:49.322483 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:49.322573 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:49.324506 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.325272 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.326648 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.327112 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.327224 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:49.329844 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:49.329850 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:49.329853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:49.329896 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:49.332787 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:49.333655 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:49.333659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:49.333662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:49.333806 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:49.334353 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:49.334361 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:49.334364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:49.334945 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:49.335327 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:49.335742 (+ 415us) server_negotiation.cc:299] Negotiation successful
0924 03:22:49.336062 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260924 03:22:49.341250 20656 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:22:49.348377 20656 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:22:49.390745 21128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:49.362119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33218 (local address 127.10.12.130:35241)
0924 03:22:49.380969 (+ 18850us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:49.380976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:49.381001 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:22:49.381085 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:49.381093 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:49.381381 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:49.381624 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:49.382724 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.384394 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.388082 (+ 3688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.389627 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.389779 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:49.389960 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:49.390202 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 03:22:49.390385 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18688,"thread_start_us":18985,"threads_started":1}
I20260924 03:22:49.390746 21130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:49.362060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:35241 (local address 127.0.0.1:33218)
0924 03:22:49.376328 (+ 14268us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:49.376358 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:49.376512 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:49.381885 (+ 5373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:49.381890 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:49.381930 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:49.382555 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:49.382569 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.384618 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.384623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:49.387816 (+ 3193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:49.387839 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.389843 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.389849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:49.390015 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:49.390044 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 03:22:49.390122 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 03:22:49.390280 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14091,"mutex_wait_us":524,"spinlock_wait_cycles":896,"thread_start_us":19466,"threads_started":1}
I20260924 03:22:49.390944 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:49.362391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34654 (local address 127.10.12.129:42053)
0924 03:22:49.380920 (+ 18529us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:49.380928 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:49.380961 (+ 33us) server_negotiation.cc:407] Connection header received
0924 03:22:49.381085 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:49.381093 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:49.381381 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:49.381639 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:49.382690 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.384407 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.388146 (+ 3739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.389255 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.389402 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:49.389661 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:49.390463 (+ 802us) server_negotiation.cc:299] Negotiation successful
0924 03:22:49.390635 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16832,"thread_start_us":17492,"threads_started":1}
I20260924 03:22:49.391532 21126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:49.362004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42053 (local address 127.0.0.1:34654)
0924 03:22:49.364014 (+ 2010us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:49.364067 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:49.364269 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:49.381868 (+ 17599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:49.381876 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:49.381913 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:49.382472 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:49.382487 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.384576 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.384582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:49.387849 (+ 3267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:49.387869 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.389542 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.389549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:49.389769 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:49.389831 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 03:22:49.389982 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 03:22:49.390243 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1676,"mutex_wait_us":297,"thread_start_us":1496,"threads_started":1}
I20260924 03:22:49.391607 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:49.364029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:44667 (local address 127.0.0.1:38224)
0924 03:22:49.381788 (+ 17759us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:49.381814 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:49.382063 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:49.382771 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:49.382775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:49.382802 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:49.383179 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:49.383191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.384676 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.384681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:49.388581 (+ 3900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:49.388598 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.390721 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.390728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:49.390922 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:49.390961 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 03:22:49.391058 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 03:22:49.391181 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17626,"mutex_wait_us":220,"spinlock_wait_cycles":7808,"thread_start_us":11954,"threads_started":1}
I20260924 03:22:49.392328 21131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:49.364614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38224 (local address 127.10.12.131:44667)
0924 03:22:49.380741 (+ 16127us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:49.380751 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:49.381930 (+ 1179us) server_negotiation.cc:407] Connection header received
0924 03:22:49.382315 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:49.382324 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:49.382593 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:49.382795 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:49.383346 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.384514 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.388789 (+ 4275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.390244 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.390365 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:49.390561 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:49.391751 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0924 03:22:49.391932 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15935,"thread_start_us":15755,"threads_started":1}
I20260924 03:22:49.395335 21048 tablet_service.cc:1467] Processing CreateTablet for tablet a6082d436bb74bdba6a1913301ecf541 (DEFAULT_TABLE table=test-table [id=4069234f2044488888dc73136b49636b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:22:49.395437 20914 tablet_service.cc:1467] Processing CreateTablet for tablet a6082d436bb74bdba6a1913301ecf541 (DEFAULT_TABLE table=test-table [id=4069234f2044488888dc73136b49636b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:22:49.395437 20780 tablet_service.cc:1467] Processing CreateTablet for tablet a6082d436bb74bdba6a1913301ecf541 (DEFAULT_TABLE table=test-table [id=4069234f2044488888dc73136b49636b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:22:49.396898 20780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6082d436bb74bdba6a1913301ecf541. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:49.396931 21048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6082d436bb74bdba6a1913301ecf541. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:49.397619 20914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6082d436bb74bdba6a1913301ecf541. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:22:49.409391 21132 tablet_bootstrap.cc:492] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854: Bootstrap starting.
I20260924 03:22:49.412184 21133 tablet_bootstrap.cc:492] T a6082d436bb74bdba6a1913301ecf541 P ce000ff670be4258892c0da863864d4a: Bootstrap starting.
I20260924 03:22:49.413017 21134 tablet_bootstrap.cc:492] T a6082d436bb74bdba6a1913301ecf541 P 2e6f98d5fc814c94b673ccb5e1a0d89e: Bootstrap starting.
I20260924 03:22:49.413161 21132 tablet_bootstrap.cc:654] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:49.414613 21132 log.cc:824] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854: Log is configured to *not* fsync() on all Append() calls
I20260924 03:22:49.416590 21134 tablet_bootstrap.cc:654] T a6082d436bb74bdba6a1913301ecf541 P 2e6f98d5fc814c94b673ccb5e1a0d89e: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:49.416594 21133 tablet_bootstrap.cc:654] T a6082d436bb74bdba6a1913301ecf541 P ce000ff670be4258892c0da863864d4a: Neither blocks nor log segments found. Creating new log.
I20260924 03:22:49.417987 21134 log.cc:824] T a6082d436bb74bdba6a1913301ecf541 P 2e6f98d5fc814c94b673ccb5e1a0d89e: Log is configured to *not* fsync() on all Append() calls
I20260924 03:22:49.417980 21133 log.cc:824] T a6082d436bb74bdba6a1913301ecf541 P ce000ff670be4258892c0da863864d4a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:22:49.418586 21132 tablet_bootstrap.cc:492] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854: No bootstrap required, opened a new log
I20260924 03:22:49.419101 21132 ts_tablet_manager.cc:1397] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260924 03:22:49.421268 21134 tablet_bootstrap.cc:492] T a6082d436bb74bdba6a1913301ecf541 P 2e6f98d5fc814c94b673ccb5e1a0d89e: No bootstrap required, opened a new log
I20260924 03:22:49.421550 21134 ts_tablet_manager.cc:1397] T a6082d436bb74bdba6a1913301ecf541 P 2e6f98d5fc814c94b673ccb5e1a0d89e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 03:22:49.422070 21133 tablet_bootstrap.cc:492] T a6082d436bb74bdba6a1913301ecf541 P ce000ff670be4258892c0da863864d4a: No bootstrap required, opened a new log
I20260924 03:22:49.422351 21133 ts_tablet_manager.cc:1397] T a6082d436bb74bdba6a1913301ecf541 P ce000ff670be4258892c0da863864d4a: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 03:22:49.428850 21134 raft_consensus.cc:348] T a6082d436bb74bdba6a1913301ecf541 P 2e6f98d5fc814c94b673ccb5e1a0d89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce000ff670be4258892c0da863864d4a" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 35241 } } peers { permanent_uuid: "972d95236f9b481aa10d3cb7c460c854" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42053 } } peers { permanent_uuid: "2e6f98d5fc814c94b673ccb5e1a0d89e" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 44667 } }
I20260924 03:22:49.429842 21134 raft_consensus.cc:374] T a6082d436bb74bdba6a1913301ecf541 P 2e6f98d5fc814c94b673ccb5e1a0d89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:49.430142 21134 raft_consensus.cc:729] T a6082d436bb74bdba6a1913301ecf541 P 2e6f98d5fc814c94b673ccb5e1a0d89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6f98d5fc814c94b673ccb5e1a0d89e, State: Initialized, Role: FOLLOWER
I20260924 03:22:49.430130 21132 raft_consensus.cc:348] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce000ff670be4258892c0da863864d4a" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 35241 } } peers { permanent_uuid: "972d95236f9b481aa10d3cb7c460c854" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42053 } } peers { permanent_uuid: "2e6f98d5fc814c94b673ccb5e1a0d89e" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 44667 } }
I20260924 03:22:49.430703 21132 raft_consensus.cc:374] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:49.430866 21132 raft_consensus.cc:729] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 972d95236f9b481aa10d3cb7c460c854, State: Initialized, Role: FOLLOWER
I20260924 03:22:49.431283 21134 consensus_queue.cc:260] T a6082d436bb74bdba6a1913301ecf541 P 2e6f98d5fc814c94b673ccb5e1a0d89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce000ff670be4258892c0da863864d4a" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 35241 } } peers { permanent_uuid: "972d95236f9b481aa10d3cb7c460c854" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42053 } } peers { permanent_uuid: "2e6f98d5fc814c94b673ccb5e1a0d89e" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 44667 } }
I20260924 03:22:49.431548 21132 consensus_queue.cc:260] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce000ff670be4258892c0da863864d4a" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 35241 } } peers { permanent_uuid: "972d95236f9b481aa10d3cb7c460c854" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42053 } } peers { permanent_uuid: "2e6f98d5fc814c94b673ccb5e1a0d89e" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 44667 } }
I20260924 03:22:49.432760 21113 heartbeater.cc:503] Master 127.10.12.190:41373 was elected leader, sending a full tablet report...
I20260924 03:22:49.433542 21134 ts_tablet_manager.cc:1428] T a6082d436bb74bdba6a1913301ecf541 P 2e6f98d5fc814c94b673ccb5e1a0d89e: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260924 03:22:49.433548 21133 raft_consensus.cc:348] T a6082d436bb74bdba6a1913301ecf541 P ce000ff670be4258892c0da863864d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce000ff670be4258892c0da863864d4a" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 35241 } } peers { permanent_uuid: "972d95236f9b481aa10d3cb7c460c854" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42053 } } peers { permanent_uuid: "2e6f98d5fc814c94b673ccb5e1a0d89e" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 44667 } }
I20260924 03:22:49.434121 21133 raft_consensus.cc:374] T a6082d436bb74bdba6a1913301ecf541 P ce000ff670be4258892c0da863864d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:22:49.434263 21133 raft_consensus.cc:729] T a6082d436bb74bdba6a1913301ecf541 P ce000ff670be4258892c0da863864d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce000ff670be4258892c0da863864d4a, State: Initialized, Role: FOLLOWER
I20260924 03:22:49.434962 21133 consensus_queue.cc:260] T a6082d436bb74bdba6a1913301ecf541 P ce000ff670be4258892c0da863864d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce000ff670be4258892c0da863864d4a" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 35241 } } peers { permanent_uuid: "972d95236f9b481aa10d3cb7c460c854" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42053 } } peers { permanent_uuid: "2e6f98d5fc814c94b673ccb5e1a0d89e" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 44667 } }
I20260924 03:22:49.435498 21132 ts_tablet_manager.cc:1428] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260924 03:22:49.437125 20979 heartbeater.cc:503] Master 127.10.12.190:41373 was elected leader, sending a full tablet report...
I20260924 03:22:49.437673 21133 ts_tablet_manager.cc:1428] T a6082d436bb74bdba6a1913301ecf541 P ce000ff670be4258892c0da863864d4a: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
W20260924 03:22:49.495332 20980 tablet.cc:2369] T a6082d436bb74bdba6a1913301ecf541 P ce000ff670be4258892c0da863864d4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:22:49.532141 21114 tablet.cc:2369] T a6082d436bb74bdba6a1913301ecf541 P 2e6f98d5fc814c94b673ccb5e1a0d89e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:22:49.616506 21139 raft_consensus.cc:482] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:22:49.616840 21139 raft_consensus.cc:504] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce000ff670be4258892c0da863864d4a" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 35241 } } peers { permanent_uuid: "972d95236f9b481aa10d3cb7c460c854" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42053 } } peers { permanent_uuid: "2e6f98d5fc814c94b673ccb5e1a0d89e" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 44667 } }
I20260924 03:22:49.618909 21139 leader_election.cc:288] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce000ff670be4258892c0da863864d4a (127.10.12.130:35241), 2e6f98d5fc814c94b673ccb5e1a0d89e (127.10.12.131:44667)
I20260924 03:22:49.627945 21128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:49.619374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:45225 (local address 127.10.12.130:35241)
0924 03:22:49.619592 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:49.619599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:49.620597 (+ 998us) server_negotiation.cc:407] Connection header received
0924 03:22:49.620753 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:49.620758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:49.620831 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:49.620903 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:49.621527 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.622653 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.625488 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.626756 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.626925 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:49.627004 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:49.627533 (+ 529us) server_negotiation.cc:299] Negotiation successful
0924 03:22:49.627674 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:22:49.628041 21131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:49.619477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:47537 (local address 127.10.12.131:44667)
0924 03:22:49.619671 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:49.619676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:49.620256 (+ 580us) server_negotiation.cc:407] Connection header received
0924 03:22:49.620480 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:49.620485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:49.620546 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:49.620634 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:22:49.621277 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.622552 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.625484 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.626732 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.626995 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:49.627063 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:49.627533 (+ 470us) server_negotiation.cc:299] Negotiation successful
0924 03:22:49.627674 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:22:49.628202 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:49.619237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:44667 (local address 127.10.12.129:47537)
0924 03:22:49.620140 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:49.620159 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:49.620330 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:49.620685 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:49.620690 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:49.620714 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:49.621080 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:49.621092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.622730 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.622736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:49.625236 (+ 2500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:49.625260 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.627060 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.627067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:49.627325 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:49.627386 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 03:22:49.627485 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 03:22:49.627708 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"mutex_wait_us":189,"thread_start_us":986,"threads_started":1}
I20260924 03:22:49.628532 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:49.619369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:35241 (local address 127.10.12.129:45225)
0924 03:22:49.620490 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:49.620505 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:22:49.620622 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:22:49.620951 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:22:49.620955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:22:49.620977 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:22:49.621322 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:49.621330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.622783 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.622787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:49.625236 (+ 2449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:22:49.625260 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.627070 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.627076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:22:49.627305 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:49.627376 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 03:22:49.627491 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 03:22:49.627701 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1026,"mutex_wait_us":71,"thread_start_us":429,"threads_started":1}
I20260924 03:22:49.629364 20934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6082d436bb74bdba6a1913301ecf541" candidate_uuid: "972d95236f9b481aa10d3cb7c460c854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce000ff670be4258892c0da863864d4a" is_pre_election: true
I20260924 03:22:49.629364 21068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6082d436bb74bdba6a1913301ecf541" candidate_uuid: "972d95236f9b481aa10d3cb7c460c854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6f98d5fc814c94b673ccb5e1a0d89e" is_pre_election: true
I20260924 03:22:49.629853 21068 raft_consensus.cc:2454] T a6082d436bb74bdba6a1913301ecf541 P 2e6f98d5fc814c94b673ccb5e1a0d89e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 972d95236f9b481aa10d3cb7c460c854 in term 0.
I20260924 03:22:49.629912 20934 raft_consensus.cc:2454] T a6082d436bb74bdba6a1913301ecf541 P ce000ff670be4258892c0da863864d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 972d95236f9b481aa10d3cb7c460c854 in term 0.
I20260924 03:22:49.630532 20734 leader_election.cc:302] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e6f98d5fc814c94b673ccb5e1a0d89e, 972d95236f9b481aa10d3cb7c460c854; no voters:
I20260924 03:22:49.631069 21139 raft_consensus.cc:2781] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:22:49.631223 21139 raft_consensus.cc:482] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:22:49.631335 21139 raft_consensus.cc:3037] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:49.633663 21139 raft_consensus.cc:504] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce000ff670be4258892c0da863864d4a" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 35241 } } peers { permanent_uuid: "972d95236f9b481aa10d3cb7c460c854" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42053 } } peers { permanent_uuid: "2e6f98d5fc814c94b673ccb5e1a0d89e" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 44667 } }
I20260924 03:22:49.634302 21139 leader_election.cc:288] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854 [CANDIDATE]: Term 1 election: Requested vote from peers ce000ff670be4258892c0da863864d4a (127.10.12.130:35241), 2e6f98d5fc814c94b673ccb5e1a0d89e (127.10.12.131:44667)
I20260924 03:22:49.634964 20934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6082d436bb74bdba6a1913301ecf541" candidate_uuid: "972d95236f9b481aa10d3cb7c460c854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce000ff670be4258892c0da863864d4a"
I20260924 03:22:49.635002 21068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6082d436bb74bdba6a1913301ecf541" candidate_uuid: "972d95236f9b481aa10d3cb7c460c854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6f98d5fc814c94b673ccb5e1a0d89e"
I20260924 03:22:49.635206 21068 raft_consensus.cc:3037] T a6082d436bb74bdba6a1913301ecf541 P 2e6f98d5fc814c94b673ccb5e1a0d89e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:49.635208 20934 raft_consensus.cc:3037] T a6082d436bb74bdba6a1913301ecf541 P ce000ff670be4258892c0da863864d4a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:22:49.638477 21068 raft_consensus.cc:2454] T a6082d436bb74bdba6a1913301ecf541 P 2e6f98d5fc814c94b673ccb5e1a0d89e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 972d95236f9b481aa10d3cb7c460c854 in term 1.
I20260924 03:22:49.638480 20934 raft_consensus.cc:2454] T a6082d436bb74bdba6a1913301ecf541 P ce000ff670be4258892c0da863864d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 972d95236f9b481aa10d3cb7c460c854 in term 1.
I20260924 03:22:49.639093 20735 leader_election.cc:302] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 972d95236f9b481aa10d3cb7c460c854, ce000ff670be4258892c0da863864d4a; no voters:
I20260924 03:22:49.639441 21139 raft_consensus.cc:2781] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:22:49.639904 21139 raft_consensus.cc:686] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854 [term 1 LEADER]: Becoming Leader. State: Replica: 972d95236f9b481aa10d3cb7c460c854, State: Running, Role: LEADER
I20260924 03:22:49.640426 21139 consensus_queue.cc:237] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce000ff670be4258892c0da863864d4a" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 35241 } } peers { permanent_uuid: "972d95236f9b481aa10d3cb7c460c854" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42053 } } peers { permanent_uuid: "2e6f98d5fc814c94b673ccb5e1a0d89e" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 44667 } }
I20260924 03:22:49.645089 20655 catalog_manager.cc:5495] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854 reported cstate change: term changed from 0 to 1, leader changed from <none> to 972d95236f9b481aa10d3cb7c460c854 (127.10.12.129). New cstate: current_term: 1 leader_uuid: "972d95236f9b481aa10d3cb7c460c854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce000ff670be4258892c0da863864d4a" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 35241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "972d95236f9b481aa10d3cb7c460c854" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6f98d5fc814c94b673ccb5e1a0d89e" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 44667 } health_report { overall_health: UNKNOWN } } }
I20260924 03:22:49.677594 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:49.672335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53106 (local address 127.10.12.129:42053)
0924 03:22:49.672560 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:49.672565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:49.672658 (+ 93us) server_negotiation.cc:407] Connection header received
0924 03:22:49.672899 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:49.672904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:49.672988 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:49.673122 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:22:49.673438 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.674210 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.675559 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.676057 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.676154 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:49.676590 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:22:49.677086 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:22:49.677192 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:49.677300 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 03:22:49.677380 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260924 03:22:49.682533 20846 tablet.cc:2369] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:22:49.686178 21068 raft_consensus.cc:1261] T a6082d436bb74bdba6a1913301ecf541 P 2e6f98d5fc814c94b673ccb5e1a0d89e [term 1 FOLLOWER]: Refusing update from remote peer 972d95236f9b481aa10d3cb7c460c854: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:22:49.686177 20934 raft_consensus.cc:1261] T a6082d436bb74bdba6a1913301ecf541 P ce000ff670be4258892c0da863864d4a [term 1 FOLLOWER]: Refusing update from remote peer 972d95236f9b481aa10d3cb7c460c854: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:22:49.687219 21143 consensus_queue.cc:1035] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6f98d5fc814c94b673ccb5e1a0d89e" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 44667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:22:49.687585 21139 consensus_queue.cc:1035] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce000ff670be4258892c0da863864d4a" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 35241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:22:49.699599 21146 mvcc.cc:205] Tried to move back new op lower bound from 7332741815024304128 to 7332741814851100672. Current Snapshot: MvccSnapshot[applied={T|T < 7332741815024304128}]
I20260924 03:22:49.716717 20655 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:22:49.717026 20655 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:22:49.721495 20655 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c54476d20b804d4093ae6dc394b72a8a: Sending DeleteTablet for 3 replicas of tablet a6082d436bb74bdba6a1913301ecf541
I20260924 03:22:49.722713 20780 tablet_service.cc:1514] Processing DeleteTablet for tablet a6082d436bb74bdba6a1913301ecf541 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:22:49 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:34654
I20260924 03:22:49.722899 20779 tablet_service.cc:1514] Processing DeleteTablet for tablet a6082d436bb74bdba6a1913301ecf541 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:22:49 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:34654
I20260924 03:22:49.723446 20914 tablet_service.cc:1514] Processing DeleteTablet for tablet a6082d436bb74bdba6a1913301ecf541 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:22:49 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:33218
I20260924 03:22:49.724153 21048 tablet_service.cc:1514] Processing DeleteTablet for tablet a6082d436bb74bdba6a1913301ecf541 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:22:49 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:38224
I20260924 03:22:49.724632 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 20717
I20260924 03:22:49.739070 21157 tablet_replica.cc:330] T a6082d436bb74bdba6a1913301ecf541 P ce000ff670be4258892c0da863864d4a: stopping tablet replica
I20260924 03:22:49.739075 21156 tablet_replica.cc:330] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854: stopping tablet replica
I20260924 03:22:49.739616 21156 raft_consensus.cc:2229] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:22:49.740249 21158 tablet_replica.cc:330] T a6082d436bb74bdba6a1913301ecf541 P 2e6f98d5fc814c94b673ccb5e1a0d89e: stopping tablet replica
I20260924 03:22:49.740681 21158 raft_consensus.cc:2229] T a6082d436bb74bdba6a1913301ecf541 P 2e6f98d5fc814c94b673ccb5e1a0d89e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:22:49.741415 21157 raft_consensus.cc:2229] T a6082d436bb74bdba6a1913301ecf541 P ce000ff670be4258892c0da863864d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:22:49.741920 21157 raft_consensus.cc:2258] T a6082d436bb74bdba6a1913301ecf541 P ce000ff670be4258892c0da863864d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:22:49.744447 21157 ts_tablet_manager.cc:1905] T a6082d436bb74bdba6a1913301ecf541 P ce000ff670be4258892c0da863864d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 03:22:49 dist-test-slave-00qs krb5kdc[20611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.129@KRBTEST.COM
I20260924 03:22:49.753147 21157 ts_tablet_manager.cc:1918] T a6082d436bb74bdba6a1913301ecf541 P ce000ff670be4258892c0da863864d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:22:49.753497 21157 log.cc:1196] T a6082d436bb74bdba6a1913301ecf541 P ce000ff670be4258892c0da863864d4a: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/a6082d436bb74bdba6a1913301ecf541
I20260924 03:22:49.754158 21157 ts_tablet_manager.cc:1939] T a6082d436bb74bdba6a1913301ecf541 P ce000ff670be4258892c0da863864d4a: Deleting consensus metadata
I20260924 03:22:49.754835 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:49.725945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53114 (local address 127.10.12.129:42053)
0924 03:22:49.726123 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:49.726130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:49.732565 (+ 6435us) server_negotiation.cc:407] Connection header received
0924 03:22:49.732878 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:49.732890 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:49.733014 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:49.733096 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:49.735123 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.736574 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.739371 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.740127 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.740240 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:49.746513 (+ 6273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:49.746533 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:49.746552 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:49.746598 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:49.750599 (+ 4001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:49.751599 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:49.751609 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:49.751613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:49.751784 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:49.752274 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:49.752279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:49.752283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:49.752747 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:49.753034 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:49.754212 (+ 1178us) server_negotiation.cc:299] Negotiation successful
0924 03:22:49.754461 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 03:22:49.755594 20840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:49.755796 21158 raft_consensus.cc:2258] T a6082d436bb74bdba6a1913301ecf541 P 2e6f98d5fc814c94b673ccb5e1a0d89e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:22:49.756686 20642 catalog_manager.cc:4841] TS ce000ff670be4258892c0da863864d4a (127.10.12.130:35241): tablet a6082d436bb74bdba6a1913301ecf541 (table test-table [id=4069234f2044488888dc73136b49636b]) successfully deleted
I20260924 03:22:49.755543 21156 raft_consensus.cc:2258] T a6082d436bb74bdba6a1913301ecf541 P 972d95236f9b481aa10d3cb7c460c854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:22:49.758520 21158 ts_tablet_manager.cc:1905] T a6082d436bb74bdba6a1913301ecf541 P 2e6f98d5fc814c94b673ccb5e1a0d89e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:22:49.765914 21158 ts_tablet_manager.cc:1918] T a6082d436bb74bdba6a1913301ecf541 P 2e6f98d5fc814c94b673ccb5e1a0d89e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:22:49.766088 21158 log.cc:1196] T a6082d436bb74bdba6a1913301ecf541 P 2e6f98d5fc814c94b673ccb5e1a0d89e: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/a6082d436bb74bdba6a1913301ecf541
I20260924 03:22:49.766497 21158 ts_tablet_manager.cc:1939] T a6082d436bb74bdba6a1913301ecf541 P 2e6f98d5fc814c94b673ccb5e1a0d89e: Deleting consensus metadata
I20260924 03:22:49.768110 20641 catalog_manager.cc:4841] TS 2e6f98d5fc814c94b673ccb5e1a0d89e (127.10.12.131:44667): tablet a6082d436bb74bdba6a1913301ecf541 (table test-table [id=4069234f2044488888dc73136b49636b]) successfully deleted
I20260924 03:22:49.859186 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 20717
I20260924 03:22:49.874956 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 20851
W20260924 03:22:49.874956 20643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.12.129:42053 (error 108)
I20260924 03:22:49.876861 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:49.876379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42053 (local address 127.0.0.1:53130)
0924 03:22:49.876566 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:49.876689 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:42053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260924 03:22:49.877089 20643 catalog_manager.cc:4584] TS 972d95236f9b481aa10d3cb7c460c854 (127.10.12.129:42053): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a6082d436bb74bdba6a1913301ecf541: Network error: Client connection negotiation failed: client connection to 127.10.12.129:42053: connect: Connection refused (error 111)
Sep 24 03:22:49 dist-test-slave-00qs krb5kdc[20611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.130@KRBTEST.COM
I20260924 03:22:49.892204 21128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:49.875848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53064 (local address 127.10.12.130:35241)
0924 03:22:49.876101 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:49.876107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:49.876160 (+ 53us) server_negotiation.cc:407] Connection header received
0924 03:22:49.876440 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:49.876445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:49.876534 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:49.876657 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:49.877985 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.878753 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.880287 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:49.880828 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:49.880993 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:49.885512 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:49.885533 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:49.885551 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:49.885601 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:49.888843 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:49.889722 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:49.889730 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:49.889732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:49.889879 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:49.890260 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:49.890263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:49.890265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:49.890696 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:49.890943 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:49.891567 (+ 624us) server_negotiation.cc:299] Negotiation successful
0924 03:22:49.891808 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260924 03:22:49.892879 20974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:49.921787 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:49.921341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42053 (local address 127.0.0.1:53136)
0924 03:22:49.921555 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:49.921649 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:42053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 03:22:49.922861 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:49.922538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42053 (local address 127.0.0.1:53140)
0924 03:22:49.922669 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:49.922750 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:42053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 03:22:49.931363 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:49.930828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42053 (local address 127.0.0.1:53142)
0924 03:22:49.931051 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:49.931145 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:42053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 03:22:49.932503 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:49.932100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42053 (local address 127.0.0.1:53154)
0924 03:22:49.932258 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:49.932380 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:42053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 03:22:49.972225 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:49.971539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42053 (local address 127.0.0.1:53170)
0924 03:22:49.971837 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:49.972009 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:42053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260924 03:22:49.973325 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:49.972948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42053 (local address 127.0.0.1:53176)
0924 03:22:49.973139 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:49.973214 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:42053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260924 03:22:49.988854 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 20851
I20260924 03:22:50.002794 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 20985
I20260924 03:22:50.008810 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:50.008372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42053 (local address 127.0.0.1:53180)
0924 03:22:50.008543 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:50.008628 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:42053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 03:22:50.009827 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:50.009441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42053 (local address 127.0.0.1:53194)
0924 03:22:50.009590 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:50.009680 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:42053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
Sep 24 03:22:50 dist-test-slave-00qs krb5kdc[20611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.131@KRBTEST.COM
I20260924 03:22:50.020921 21131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:50.003993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37676 (local address 127.10.12.131:44667)
0924 03:22:50.004336 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0924 03:22:50.004343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:22:50.004373 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:22:50.004538 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:22:50.004544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:22:50.004679 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:22:50.004822 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:22:50.006705 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:50.007583 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:50.009271 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:22:50.009804 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:22:50.010025 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:22:50.014608 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:22:50.014627 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:22:50.014643 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:22:50.014686 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:22:50.017595 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:50.018500 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:50.018510 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:50.018512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:50.018681 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:22:50.019143 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:22:50.019148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:22:50.019152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:22:50.019616 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:22:50.019877 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 03:22:50.020422 (+ 545us) server_negotiation.cc:299] Negotiation successful
0924 03:22:50.020612 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260924 03:22:50.021622 21108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:50.074503 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:50.073997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42053 (local address 127.0.0.1:53204)
0924 03:22:50.074234 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:50.074321 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:42053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260924 03:22:50.075488 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:50.075141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42053 (local address 127.0.0.1:53210)
0924 03:22:50.075281 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:50.075350 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:42053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 03:22:50.112609 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:50.112024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42053 (local address 127.0.0.1:53222)
0924 03:22:50.112250 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:50.112374 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:42053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 03:22:50.112939 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 20985
I20260924 03:22:50.114530 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:22:50.114079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42053 (local address 127.0.0.1:53234)
0924 03:22:50.114252 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 03:22:50.114356 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:42053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 03:22:50.126127 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 20624
I20260924 03:22:50.127259 20686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:22:50.237870 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 20624
2026-09-24T03:22:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4460 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:22:50 dist-test-slave-00qs krb5kdc[21167](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:22:50 dist-test-slave-00qs krb5kdc[21167](info): set up 2 sockets
Sep 24 03:22:50 dist-test-slave-00qs krb5kdc[21167](info): commencing operation
krb5kdc: starting...
W20260924 03:22:51.268980 10290 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:22:51 dist-test-slave-00qs krb5kdc[21167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:22:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:22:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.12.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.12.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.12.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/rangeradmin_127.10.12.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.12.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/rangeradmin_127.10.12.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.12.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.12.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.12.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/rangerlookup_127.10.12.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.12.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/rangerlookup_127.10.12.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.12.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.12.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.12.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/HTTP_127.10.12.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.12.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/HTTP_127.10.12.148@KRBTEST.COM.keytab.
I20260924 03:22:51.608294 10290 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskxoV79t/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/postgres -l logfile start
2026-09-24 03:22:54.579 UTC [21197] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 03:22:54.579 UTC [21197] LOG: listening on IPv4 address "127.10.12.148", port 55411
2026-09-24 03:22:54.584 UTC [21197] LOG: listening on Unix socket "/tmp/.s.PGSQL.55411"
2026-09-24 03:22:54.609 UTC [21199] LOG: database system was shut down at 2026-09-24 03:22:52 UTC
2026-09-24 03:22:54.614 UTC [21197] LOG: database system is ready to accept connections
I20260924 03:22:55.528690 10290 mini_postgres.cc:96] Postgres bound to 55411
2026-09-24 03:22:55.547 UTC [21208] FATAL: database "slave" does not exist
127.10.12.148:55411 - accepting connections
I20260924 03:22:55.547917 10290 mini_ranger.cc:161] Starting Ranger...
I20260924 03:22:55.578938 10290 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 03:22:56.172875 10290 mini_ranger.cc:87] Created ranger Postgres database
I20260924 03:22:56.173019 10290 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ranger-admin
2026-09-24 03:22:56,479 [I] DB FLAVOR :POSTGRES
2026-09-24 03:22:56,479 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 03:22:56,479 [I] Checking connection..
2026-09-24 03:22:56,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:55411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 03:22:56,720 [I] Checking connection passed.
2026-09-24 03:22:56,720 [I] --------- Verifying version history table ---------
2026-09-24 03:22:56,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:55411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:22:56,994 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 03:22:56,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:55411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:22:57,277 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 03:22:57,277 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 03:22:57,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:55411/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 03:22:57.521 UTC [21272] WARNING: there is no transaction in progress
2026-09-24 03:22:57,528 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 03:22:57,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:55411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:22:57,807 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 03:22:57,808 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 03:22:57,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:55411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 03:22:58,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:55411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 03:22:58,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:55411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-00qs.c.gcp-upstream.internal','N') ;"
2026-09-24 03:22:58,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:55411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 03:22:58,847 [I] Table x_portal_user does not exist in database ranger
2026-09-24 03:22:58,847 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:55411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 03:22:59,114 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 03:22:59,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:55411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 03:22:59,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:55411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 03:22:59,645 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 03:22:59,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:55411/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 03:23:00.800 UTC [21380] WARNING: there is no transaction in progress
2026-09-24 03:23:00.819 UTC [21380] WARNING: there is no transaction in progress
2026-09-24 03:23:00.839 UTC [21380] WARNING: there is no transaction in progress
2026-09-24 03:23:00.858 UTC [21380] WARNING: there is no transaction in progress
2026-09-24 03:23:00.878 UTC [21380] WARNING: there is no transaction in progress
2026-09-24 03:23:00.900 UTC [21380] WARNING: there is no transaction in progress
2026-09-24 03:23:00.964 UTC [21380] WARNING: there is no transaction in progress
2026-09-24 03:23:00.972 UTC [21380] WARNING: there is no transaction in progress
2026-09-24 03:23:00.991 UTC [21380] WARNING: there is no transaction in progress
2026-09-24 03:23:01.003 UTC [21380] WARNING: there is no transaction in progress
2026-09-24 03:23:01.017 UTC [21380] WARNING: there is no transaction in progress
2026-09-24 03:23:01.034 UTC [21380] WARNING: there is no transaction in progress
2026-09-24 03:23:01.046 UTC [21380] WARNING: there is no transaction in progress
2026-09-24 03:23:01.054 UTC [21380] WARNING: there is no transaction in progress
2026-09-24 03:23:01.063 UTC [21380] WARNING: there is no transaction in progress
2026-09-24 03:23:01.071 UTC [21380] WARNING: there is no transaction in progress
2026-09-24 03:23:01,947 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 03:23:01,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:55411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 03:23:02,214 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 03:23:02,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:55411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 03:23:02,481 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 03:23:02,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:55411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 03:23:02,746 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 03:23:02,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:55411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 03:23:03,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:55411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 03:23:03,289 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:55411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-00qs.c.gcp-upstream.internal';"
2026-09-24 03:23:03,556 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 03:23:03,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:55411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 03:23:03,829 [I] DB_PATCHES have already been applied
I20260924 03:23:03.834937 10290 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ranger-admin:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskxoV79t/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 03:23:03.835047 10290 mini_ranger.cc:193] Using host: 127.10.12.148
I20260924 03:23:03.850031 10290 mini_ranger.cc:227] Ranger admin URL: http://127.10.12.148:54685
Sep 24, 2026 3:23:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:23:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:23:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 3:23:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 3:23:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 3:23:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 3:23:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 03:23:04.919088 10290 mini_ranger.cc:160] Time spent starting Ranger: real 9.371s user 0.000s sys 0.047s
Sep 24, 2026 3:23:27 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 3:23:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 3:23:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 3:23:27 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 3:23:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 3:23:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 3:23:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 3:23:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 3:23:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 3:23:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 3:23:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 3:23:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 3:23:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 3:23:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 3:23:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 3:23:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 3:23:27 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 3:23:28 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 03:23:28.874409 10290 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.12.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.12.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.12.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/rangerkms_127.10.12.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.12.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/rangerkms_127.10.12.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.12.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.12.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.12.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/HTTP_127.10.12.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.12.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/HTTP_127.10.12.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 03:23:29 dist-test-slave-00qs krb5kdc[21167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220209, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 03:23:29.056514 10290 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 03:23:29.077632 10290 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 03:23:30.251070 10290 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 03:23:30.251217 10290 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ranger-kms
I20260924 03:23:30.251331 10290 mini_ranger_kms.cc:227] Using postgres at 127.10.12.148:55411
2026-09-24 03:23:30,660 [I] DB FLAVOR :POSTGRES
2026-09-24 03:23:30,660 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 03:23:30,661 [I] Checking connection
2026-09-24 03:23:30,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:55411/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 03:23:30,908 [I] connection success
2026-09-24 03:23:30,908 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 03:23:30,908 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 03:23:30,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:55411/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 03:23:31,154 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 03:23:31,154 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 03:23:31,154 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 03:23:31,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:55411/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 03:23:31,421 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 03:23:31.970278 10290 mini_ranger_kms.cc:310] Created kms service
I20260924 03:23:32.137944 10290 mini_ranger_kms.cc:326] Created kudu user
I20260924 03:23:32.210177 10290 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 03:23:32 dist-test-slave-00qs krb5kdc[21167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220209, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.12.148@KRBTEST.COM
I20260924 03:23:32.673038 10290 mini_ranger_kms.cc:384] Added ranger policy
I20260924 03:23:32.673348 10290 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ranger-kms:/tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 03:23:32.673395 10290 mini_ranger_kms.cc:240] Using host: 127.10.12.148
I20260924 03:23:32.699914 10290 mini_ranger_kms.cc:276] Ranger KMS PID: 21561
I20260924 03:23:32.700126 10290 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.12.148:55893
Sep 24, 2026 3:23:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:23:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:23:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 3:23:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 3:23:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 3:23:34 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 03:23:34.808470 10290 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.752s user 0.125s sys 0.310s
I20260924 03:23:34.808714 10290 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 03:23:34.808917 10290 mini_ranger_kms.cc:401] 127.10.12.148:55893/kms/v1/keys
Sep 24 03:23:34 dist-test-slave-00qs krb5kdc[21167](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790220214, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.12.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 03:23:35 dist-test-slave-00qs krb5kdc[21167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220214, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.12.148@KRBTEST.COM for HTTP/127.10.12.148@KRBTEST.COM
Sep 24 03:23:35 dist-test-slave-00qs krb5kdc[21167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220214, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.12.148@KRBTEST.COM for HTTP/127.10.12.148@KRBTEST.COM
Sep 24 03:23:38 dist-test-slave-00qs krb5kdc[21167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:23:38.344653 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:44627
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:44627
--ranger_config_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.12.148:55893/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:23:38.701748 21611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:23:38.702054 21611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:23:38.702174 21611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:23:38.710872 21611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:23:38.711016 21611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:23:38.711061 21611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:23:38.711113 21611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:23:38.711177 21611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:23:38.722502 21611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.12.148:55893/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:44627
--ranger_config_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:44627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:23:38.723866 21611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:23:38.725247 21611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:23:38.737531 21616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:23:38.737769 21617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:23:38.737804 21619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:23:38.739145 21611 server_base.cc:1034] running on GCE node
I20260924 03:23:38.740054 21611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:23:38.741572 21611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:23:38.742825 21611 hybrid_clock.cc:648] HybridClock initialized: now 1790220218742769 us; error 55 us; skew 500 ppm
I20260924 03:23:38.743254 21611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:23:38 dist-test-slave-00qs krb5kdc[21167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220218, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:23:38.754457 21611 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:23:38.756263 21611 webserver.cc:460] Webserver started at http://127.10.12.190:42815/ using document root <none> and password file <none>
I20260924 03:23:38.756965 21611 fs_manager.cc:359] Metadata directory not provided
I20260924 03:23:38.757081 21611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:23:38.757381 21611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:23:38 dist-test-slave-00qs krb5kdc[21167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220218, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for HTTP/127.10.12.148@KRBTEST.COM
I20260924 03:23:38.819425 21611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "9086dd7395cd41da83f7d5bf11b21d22"
format_stamp: "Formatted at 2026-09-24 03:23:38 on dist-test-slave-00qs"
server_key: "bfb2775f7db6c439ea0956bb39e72c02"
server_key_iv: "966924a1d61a6a8579550e052bcdfb7a"
server_key_version: "kuduclusterkey@0"
I20260924 03:23:38.821213 21611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "9086dd7395cd41da83f7d5bf11b21d22"
format_stamp: "Formatted at 2026-09-24 03:23:38 on dist-test-slave-00qs"
server_key: "bfb2775f7db6c439ea0956bb39e72c02"
server_key_iv: "966924a1d61a6a8579550e052bcdfb7a"
server_key_version: "kuduclusterkey@0"
I20260924 03:23:38.826916 21611 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:23:38.847538 21626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:23:38.849097 21611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:23:38.849268 21611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "9086dd7395cd41da83f7d5bf11b21d22"
format_stamp: "Formatted at 2026-09-24 03:23:38 on dist-test-slave-00qs"
server_key: "bfb2775f7db6c439ea0956bb39e72c02"
server_key_iv: "966924a1d61a6a8579550e052bcdfb7a"
server_key_version: "kuduclusterkey@0"
I20260924 03:23:38.849498 21611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:23:38.865561 21611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:23:38.869913 21611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:23:38.870203 21611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:23:38.889021 21611 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:44627
I20260924 03:23:38.889047 21677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:44627 every 8 connection(s)
I20260924 03:23:38.890383 21611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:23:38.893249 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 21611
I20260924 03:23:38.893450 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:23:38.894069 10290 external_mini_cluster.cc:1428] Setting key 95985d75579cee13c0237c9113cd0628
I20260924 03:23:38.894430 21678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:23:38.909765 21678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9086dd7395cd41da83f7d5bf11b21d22: Bootstrap starting.
Sep 24 03:23:38 dist-test-slave-00qs krb5kdc[21167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:23:38.914050 21678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9086dd7395cd41da83f7d5bf11b21d22: Neither blocks nor log segments found. Creating new log.
I20260924 03:23:38.915444 21678 log.cc:824] T 00000000000000000000000000000000 P 9086dd7395cd41da83f7d5bf11b21d22: Log is configured to *not* fsync() on all Append() calls
I20260924 03:23:38.918373 21678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9086dd7395cd41da83f7d5bf11b21d22: No bootstrap required, opened a new log
I20260924 03:23:38.919483 21681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:23:38.895943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52454 (local address 127.10.12.190:44627)
0924 03:23:38.896747 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0924 03:23:38.896762 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:23:38.896809 (+ 47us) server_negotiation.cc:407] Connection header received
0924 03:23:38.899129 (+ 2320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:23:38.899162 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:23:38.899739 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:23:38.900263 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:23:38.901518 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:38.903608 (+ 2090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:38.905352 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:38.906510 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:38.906734 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:23:38.911147 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:23:38.911159 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:23:38.911179 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:23:38.911238 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:23:38.915260 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:23:38.916134 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:23:38.916144 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:23:38.916151 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:23:38.916347 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:23:38.916724 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:23:38.916728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:23:38.916730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:23:38.917290 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:23:38.917641 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0924 03:23:38.918053 (+ 412us) server_negotiation.cc:299] Negotiation successful
0924 03:23:38.918464 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":310,"threads_started":1}
I20260924 03:23:38.927713 21678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9086dd7395cd41da83f7d5bf11b21d22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9086dd7395cd41da83f7d5bf11b21d22" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44627 } }
I20260924 03:23:38.928158 21678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9086dd7395cd41da83f7d5bf11b21d22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:23:38.928256 21678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9086dd7395cd41da83f7d5bf11b21d22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9086dd7395cd41da83f7d5bf11b21d22, State: Initialized, Role: FOLLOWER
I20260924 03:23:38.928862 21678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9086dd7395cd41da83f7d5bf11b21d22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9086dd7395cd41da83f7d5bf11b21d22" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44627 } }
I20260924 03:23:38.929165 21678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9086dd7395cd41da83f7d5bf11b21d22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:23:38.929309 21678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9086dd7395cd41da83f7d5bf11b21d22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:23:38.929491 21678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9086dd7395cd41da83f7d5bf11b21d22 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:23:38.932008 21678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9086dd7395cd41da83f7d5bf11b21d22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9086dd7395cd41da83f7d5bf11b21d22" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44627 } }
I20260924 03:23:38.932544 21678 leader_election.cc:302] T 00000000000000000000000000000000 P 9086dd7395cd41da83f7d5bf11b21d22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9086dd7395cd41da83f7d5bf11b21d22; no voters:
I20260924 03:23:38.933136 21678 leader_election.cc:288] T 00000000000000000000000000000000 P 9086dd7395cd41da83f7d5bf11b21d22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:23:38.933257 21683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9086dd7395cd41da83f7d5bf11b21d22 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:23:38.934002 21683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9086dd7395cd41da83f7d5bf11b21d22 [term 1 LEADER]: Becoming Leader. State: Replica: 9086dd7395cd41da83f7d5bf11b21d22, State: Running, Role: LEADER
I20260924 03:23:38.934592 21683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9086dd7395cd41da83f7d5bf11b21d22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9086dd7395cd41da83f7d5bf11b21d22" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44627 } }
I20260924 03:23:38.935326 21678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9086dd7395cd41da83f7d5bf11b21d22 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:23:38.937839 21685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9086dd7395cd41da83f7d5bf11b21d22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9086dd7395cd41da83f7d5bf11b21d22. Latest consensus state: current_term: 1 leader_uuid: "9086dd7395cd41da83f7d5bf11b21d22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9086dd7395cd41da83f7d5bf11b21d22" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44627 } } }
I20260924 03:23:38.937947 21684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9086dd7395cd41da83f7d5bf11b21d22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9086dd7395cd41da83f7d5bf11b21d22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9086dd7395cd41da83f7d5bf11b21d22" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44627 } } }
I20260924 03:23:38.938179 21685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9086dd7395cd41da83f7d5bf11b21d22 [sys.catalog]: This master's current role is: LEADER
I20260924 03:23:38.938244 21684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9086dd7395cd41da83f7d5bf11b21d22 [sys.catalog]: This master's current role is: LEADER
I20260924 03:23:38.939002 21691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:23:38.945374 21691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:23:38.946020 21678 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 03:23:38.969304 21691 catalog_manager.cc:1347] Generated new cluster ID: 6f2ed9a5ed174944b822f923f748e285
I20260924 03:23:38.969475 21691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:23:38.979151 21691 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:23:38.980808 21691 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:23:38.995458 21691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9086dd7395cd41da83f7d5bf11b21d22: Generated new TSK 0
I20260924 03:23:38.996627 21691 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
I20260924 03:23:39.780251 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.12.148:55893/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:44627
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:23:40 dist-test-slave-00qs krb5kdc[21167](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790220220, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 03:23:40.222630 21723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:23:40.223069 21723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:23:40.223284 21723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:23:40.231180 21723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:23:40.231343 21723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:23:40.231531 21723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:23:40.243098 21723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.12.148:55893/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:44627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:23:40.244544 21723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:23:40.245867 21723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:23:40.265658 21733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:23:40.265626 21734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:23:40.267220 21736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:23:40.268380 21723 server_base.cc:1034] running on GCE node
I20260924 03:23:40.269237 21723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:23:40.270655 21723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:23:40.271989 21723 hybrid_clock.cc:648] HybridClock initialized: now 1790220220271865 us; error 99 us; skew 500 ppm
I20260924 03:23:40.272454 21723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:23:40 dist-test-slave-00qs krb5kdc[21167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220220, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:23:40.277046 21723 init.cc:377] Logged in from keytab as kudu/127.10.12.129@KRBTEST.COM (short username kudu)
I20260924 03:23:40.289935 21723 webserver.cc:460] Webserver started at http://127.10.12.129:45057/ using document root <none> and password file <none>
I20260924 03:23:40.291154 21723 fs_manager.cc:359] Metadata directory not provided
I20260924 03:23:40.291498 21723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:23:40.292217 21723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:23:40 dist-test-slave-00qs krb5kdc[21167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220220, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for HTTP/127.10.12.148@KRBTEST.COM
I20260924 03:23:40.318519 21723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "0403757537c44724a898dd47d7f552a2"
format_stamp: "Formatted at 2026-09-24 03:23:40 on dist-test-slave-00qs"
server_key: "89d0703e64610eb08cdc94796c1f40d7"
server_key_iv: "7467dc7f9bfaf5848ab5d5363dbe4317"
server_key_version: "kuduclusterkey@0"
I20260924 03:23:40.319772 21723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "0403757537c44724a898dd47d7f552a2"
format_stamp: "Formatted at 2026-09-24 03:23:40 on dist-test-slave-00qs"
server_key: "89d0703e64610eb08cdc94796c1f40d7"
server_key_iv: "7467dc7f9bfaf5848ab5d5363dbe4317"
server_key_version: "kuduclusterkey@0"
I20260924 03:23:40.340461 21723 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.012s sys 0.005s
I20260924 03:23:40.372619 21743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:23:40.378479 21723 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.001s sys 0.007s
I20260924 03:23:40.378877 21723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "0403757537c44724a898dd47d7f552a2"
format_stamp: "Formatted at 2026-09-24 03:23:40 on dist-test-slave-00qs"
server_key: "89d0703e64610eb08cdc94796c1f40d7"
server_key_iv: "7467dc7f9bfaf5848ab5d5363dbe4317"
server_key_version: "kuduclusterkey@0"
I20260924 03:23:40.379315 21723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:23:40.428787 21723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:23:40.432993 21723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:23:40.433310 21723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:23:40.434382 21723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:23:40.436594 21723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:23:40.436684 21723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:23:40.436810 21723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:23:40.436874 21723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:23:40.516332 21723 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:41639
I20260924 03:23:40.516350 21855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:41639 every 8 connection(s)
I20260924 03:23:40.518199 21723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:23:40.521881 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 21723
I20260924 03:23:40.522089 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:23:40.522883 10290 external_mini_cluster.cc:1428] Setting key a3fa5a144e4b249aa6f6be5346356afd
Sep 24 03:23:40 dist-test-slave-00qs krb5kdc[21167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220220, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.130@KRBTEST.COM" created.
I20260924 03:23:40.578545 21859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:23:40.526627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:44627 (local address 127.10.12.129:47037)
0924 03:23:40.536514 (+ 9887us) negotiation.cc:107] Waiting for socket to connect
0924 03:23:40.536584 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 03:23:40.539549 (+ 2965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:23:40.549023 (+ 9474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:23:40.549039 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:23:40.549689 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:23:40.550531 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:23:40.550567 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:23:40.552644 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:23:40.552650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:23:40.554690 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:23:40.554711 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:23:40.556270 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:23:40.556291 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:23:40.556624 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:23:40.557499 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:23:40.557535 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:23:40.560556 (+ 3021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:23:40.565506 (+ 4950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:23:40.565515 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:23:40.565565 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:23:40.566339 (+ 774us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:23:40.567014 (+ 675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:23:40.567019 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:23:40.567023 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:23:40.567305 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:23:40.575157 (+ 7852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:23:40.575168 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:23:40.575721 (+ 553us) client_negotiation.cc:769] Sending connection context
0924 03:23:40.576091 (+ 370us) client_negotiation.cc:240] Negotiation successful
0924 03:23:40.576666 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9431,"thread_start_us":10187,"threads_started":1}
I20260924 03:23:40.580075 21858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:23:40.525792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:47037 (local address 127.10.12.190:44627)
0924 03:23:40.548184 (+ 22392us) server_negotiation.cc:206] Beginning negotiation
0924 03:23:40.548190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:23:40.548212 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:23:40.548279 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:23:40.548285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:23:40.548389 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:23:40.548519 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:23:40.550815 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:40.552406 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:40.555072 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:40.555745 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:40.555892 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:23:40.560938 (+ 5046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:23:40.560947 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:23:40.560952 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:23:40.561007 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:23:40.565289 (+ 4282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:23:40.566483 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:23:40.566489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:23:40.566493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:23:40.566665 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:23:40.574337 (+ 7672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:23:40.574345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:23:40.574349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:23:40.574905 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:23:40.579243 (+ 4338us) server_negotiation.cc:1035] Waiting for connection context
0924 03:23:40.579433 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 03:23:40.579686 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22245,"thread_start_us":26161,"threads_started":1}
I20260924 03:23:40.581581 21856 heartbeater.cc:347] Connected to a master server at 127.10.12.190:44627
I20260924 03:23:40.581971 21856 heartbeater.cc:464] Registering TS with master...
I20260924 03:23:40.582935 21856 heartbeater.cc:511] Master 127.10.12.190:44627 requested a full tablet report, sending...
I20260924 03:23:40.585660 21643 ts_manager.cc:194] Registered new tserver with Master: 0403757537c44724a898dd47d7f552a2 (127.10.12.129:41639)
I20260924 03:23:40.587231 21643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.129@KRBTEST.COM'} at 127.10.12.129:47037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
I20260924 03:23:40.605770 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.12.148:55893/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:44627
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:23:40 dist-test-slave-00qs krb5kdc[21167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220220, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for HTTP/127.10.12.148@KRBTEST.COM
Sep 24 03:23:40 dist-test-slave-00qs krb5kdc[21167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220220, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for HTTP/127.10.12.148@KRBTEST.COM
W20260924 03:23:41.113399 21862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:23:41.113904 21862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:23:41.114102 21862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:23:41.126009 21862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:23:41.126250 21862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:23:41.126482 21862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:23:41.140532 21862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.12.148:55893/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:44627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:23:41.142277 21862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:23:41.143607 21862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:23:41.183555 21870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:23:41.186055 21869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:23:41.187155 21872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:23:41.188071 21862 server_base.cc:1034] running on GCE node
I20260924 03:23:41.189100 21862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:23:41.196311 21862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:23:41.197712 21862 hybrid_clock.cc:648] HybridClock initialized: now 1790220221197611 us; error 104 us; skew 500 ppm
I20260924 03:23:41.198140 21862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:23:41 dist-test-slave-00qs krb5kdc[21167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220221, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:23:41.202473 21862 init.cc:377] Logged in from keytab as kudu/127.10.12.130@KRBTEST.COM (short username kudu)
I20260924 03:23:41.212045 21862 webserver.cc:460] Webserver started at http://127.10.12.130:38271/ using document root <none> and password file <none>
I20260924 03:23:41.213272 21862 fs_manager.cc:359] Metadata directory not provided
I20260924 03:23:41.213541 21862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:23:41.213893 21862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:23:41 dist-test-slave-00qs krb5kdc[21167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220221, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for HTTP/127.10.12.148@KRBTEST.COM
I20260924 03:23:41.239157 21862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "d6558bed5c50498e9830889879192658"
format_stamp: "Formatted at 2026-09-24 03:23:41 on dist-test-slave-00qs"
server_key: "2767211d4d2e2489262dda147018cf43"
server_key_iv: "b3e7bdf9c655d677999e5f6f091ea29c"
server_key_version: "kuduclusterkey@0"
I20260924 03:23:41.240136 21862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "d6558bed5c50498e9830889879192658"
format_stamp: "Formatted at 2026-09-24 03:23:41 on dist-test-slave-00qs"
server_key: "2767211d4d2e2489262dda147018cf43"
server_key_iv: "b3e7bdf9c655d677999e5f6f091ea29c"
server_key_version: "kuduclusterkey@0"
I20260924 03:23:41.245646 21862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 03:23:41.269905 21879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:23:41.272117 21862 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.003s
I20260924 03:23:41.272428 21862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "d6558bed5c50498e9830889879192658"
format_stamp: "Formatted at 2026-09-24 03:23:41 on dist-test-slave-00qs"
server_key: "2767211d4d2e2489262dda147018cf43"
server_key_iv: "b3e7bdf9c655d677999e5f6f091ea29c"
server_key_version: "kuduclusterkey@0"
I20260924 03:23:41.272809 21862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:23:41.309233 21862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:23:41.313870 21862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:23:41.314143 21862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:23:41.315090 21862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:23:41.316529 21862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:23:41.317001 21862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:23:41.317107 21862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:23:41.317150 21862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:23:41.351298 21862 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:38327
I20260924 03:23:41.351329 21998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:38327 every 8 connection(s)
I20260924 03:23:41.352596 21862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:23:41.359031 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 21862
I20260924 03:23:41.359184 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:23:41.359639 10290 external_mini_cluster.cc:1428] Setting key 0d4d0b3767040ea30c07f03e5a32e569
Sep 24 03:23:41 dist-test-slave-00qs krb5kdc[21167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220221, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:23:41.375279 22001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:23:41.355199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:57699 (local address 127.10.12.190:44627)
0924 03:23:41.355710 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0924 03:23:41.355717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:23:41.356665 (+ 948us) server_negotiation.cc:407] Connection header received
0924 03:23:41.358487 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:23:41.358492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:23:41.358589 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:23:41.358724 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:23:41.361066 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:41.361970 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:41.363817 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:41.364298 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:41.364447 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:23:41.368659 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:23:41.368664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:23:41.368667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:23:41.368704 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:23:41.371890 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:23:41.372760 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:23:41.372766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:23:41.372770 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:23:41.372920 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:23:41.373356 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:23:41.373361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:23:41.373364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:23:41.373719 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:23:41.373857 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 03:23:41.374630 (+ 773us) server_negotiation.cc:299] Negotiation successful
0924 03:23:41.374887 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":262,"threads_started":1}
I20260924 03:23:41.376225 22002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:23:41.355737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:44627 (local address 127.10.12.130:57699)
0924 03:23:41.356505 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0924 03:23:41.356544 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 03:23:41.358173 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:23:41.359135 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:23:41.359147 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:23:41.359633 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:23:41.360878 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:23:41.360904 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:23:41.362139 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:23:41.362143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:23:41.363634 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:23:41.363650 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:23:41.364604 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:23:41.364613 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:23:41.364974 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:23:41.365735 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:23:41.365765 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:23:41.368375 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:23:41.372094 (+ 3719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:23:41.372102 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:23:41.372119 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:23:41.372621 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:23:41.373049 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:23:41.373053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:23:41.373055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:23:41.373253 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:23:41.373868 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:23:41.373876 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:23:41.374244 (+ 368us) client_negotiation.cc:769] Sending connection context
0924 03:23:41.374486 (+ 242us) client_negotiation.cc:240] Negotiation successful
0924 03:23:41.374969 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":311,"threads_started":1}
I20260924 03:23:41.377929 21999 heartbeater.cc:347] Connected to a master server at 127.10.12.190:44627
I20260924 03:23:41.378312 21999 heartbeater.cc:464] Registering TS with master...
I20260924 03:23:41.379246 21999 heartbeater.cc:511] Master 127.10.12.190:44627 requested a full tablet report, sending...
I20260924 03:23:41.381093 21643 ts_manager.cc:194] Registered new tserver with Master: d6558bed5c50498e9830889879192658 (127.10.12.130:38327)
I20260924 03:23:41.381955 21643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.130@KRBTEST.COM'} at 127.10.12.130:57699
WARNING: no policy specified for kudu/127.10.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
I20260924 03:23:41.415824 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.12.148:55893/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:44627
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:23:41.591432 21856 heartbeater.cc:503] Master 127.10.12.190:44627 was elected leader, sending a full tablet report...
W20260924 03:23:41.761965 22005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:23:41.762275 22005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:23:41.762390 22005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:23:41.770290 22005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:23:41.770437 22005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:23:41.770596 22005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:23:41.780845 22005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.12.148:55893/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:44627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:23:41.782164 22005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:23:41.783622 22005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:23:41.797679 22010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:23:41.797744 22011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:23:41.797701 22013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:23:41.798475 22005 server_base.cc:1034] running on GCE node
I20260924 03:23:41.799103 22005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:23:41.800165 22005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:23:41.801421 22005 hybrid_clock.cc:648] HybridClock initialized: now 1790220221801360 us; error 52 us; skew 500 ppm
I20260924 03:23:41.801764 22005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:23:41 dist-test-slave-00qs krb5kdc[21167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220221, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:23:41.805147 22005 init.cc:377] Logged in from keytab as kudu/127.10.12.131@KRBTEST.COM (short username kudu)
I20260924 03:23:41.807212 22005 webserver.cc:460] Webserver started at http://127.10.12.131:37345/ using document root <none> and password file <none>
I20260924 03:23:41.807987 22005 fs_manager.cc:359] Metadata directory not provided
I20260924 03:23:41.808110 22005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:23:41.808408 22005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:23:41 dist-test-slave-00qs krb5kdc[21167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220221, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for HTTP/127.10.12.148@KRBTEST.COM
I20260924 03:23:41.824753 22005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "26227ae0411048fe84b8ca66d37eab65"
format_stamp: "Formatted at 2026-09-24 03:23:41 on dist-test-slave-00qs"
server_key: "780f8cc54329c6d68c5cc5820774226a"
server_key_iv: "e3df095a7731b31e4d6eefe514b3e762"
server_key_version: "kuduclusterkey@0"
I20260924 03:23:41.825939 22005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "26227ae0411048fe84b8ca66d37eab65"
format_stamp: "Formatted at 2026-09-24 03:23:41 on dist-test-slave-00qs"
server_key: "780f8cc54329c6d68c5cc5820774226a"
server_key_iv: "e3df095a7731b31e4d6eefe514b3e762"
server_key_version: "kuduclusterkey@0"
I20260924 03:23:41.833806 22005 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260924 03:23:41.852798 22020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:23:41.854907 22005 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260924 03:23:41.855115 22005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "26227ae0411048fe84b8ca66d37eab65"
format_stamp: "Formatted at 2026-09-24 03:23:41 on dist-test-slave-00qs"
server_key: "780f8cc54329c6d68c5cc5820774226a"
server_key_iv: "e3df095a7731b31e4d6eefe514b3e762"
server_key_version: "kuduclusterkey@0"
I20260924 03:23:41.855324 22005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:23:41.871307 22005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:23:41.875419 22005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:23:41.875689 22005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:23:41.876940 22005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:23:41.878743 22005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:23:41.878840 22005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:23:41.878965 22005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:23:41.879020 22005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:23:41.915176 22005 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:46595
I20260924 03:23:41.915210 22132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:46595 every 8 connection(s)
I20260924 03:23:41.916865 22005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:23:41.920164 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 22005
I20260924 03:23:41.920353 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:23:41.920902 10290 external_mini_cluster.cc:1428] Setting key 5225a6ef6903ecfca676efa82d5e0840
Sep 24 03:23:41 dist-test-slave-00qs krb5kdc[21167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220221, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:23:41.938401 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:23:41.919594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:34065 (local address 127.10.12.190:44627)
0924 03:23:41.920103 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0924 03:23:41.920108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:23:41.921134 (+ 1026us) server_negotiation.cc:407] Connection header received
0924 03:23:41.923353 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:23:41.923360 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:23:41.923557 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:23:41.923654 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:23:41.925354 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:41.926093 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:41.927721 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:41.928228 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:41.928337 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:23:41.932331 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:23:41.932337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:23:41.932341 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:23:41.932380 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:23:41.934884 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:23:41.935860 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:23:41.935866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:23:41.935870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:23:41.936047 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:23:41.936587 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:23:41.936592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:23:41.936596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:23:41.936942 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:23:41.937070 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 03:23:41.937797 (+ 727us) server_negotiation.cc:299] Negotiation successful
0924 03:23:41.938041 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":226,"threads_started":1}
I20260924 03:23:41.939415 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:23:41.920202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:44627 (local address 127.10.12.131:34065)
0924 03:23:41.920971 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0924 03:23:41.921017 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 03:23:41.922890 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:23:41.923920 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:23:41.923930 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:23:41.924557 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:23:41.925184 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:23:41.925203 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:23:41.926243 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:23:41.926247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:23:41.927576 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:23:41.927589 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:23:41.928450 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:23:41.928458 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:23:41.928686 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:23:41.929827 (+ 1141us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:23:41.929855 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:23:41.932076 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:23:41.935070 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:23:41.935081 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:23:41.935102 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:23:41.935734 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:23:41.936156 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:23:41.936161 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:23:41.936165 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:23:41.936485 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:23:41.937061 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:23:41.937069 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:23:41.937423 (+ 354us) client_negotiation.cc:769] Sending connection context
0924 03:23:41.937678 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 03:23:41.938223 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":322,"threads_started":1}
I20260924 03:23:41.940902 22133 heartbeater.cc:347] Connected to a master server at 127.10.12.190:44627
I20260924 03:23:41.941217 22133 heartbeater.cc:464] Registering TS with master...
I20260924 03:23:41.941946 22133 heartbeater.cc:511] Master 127.10.12.190:44627 requested a full tablet report, sending...
I20260924 03:23:41.943496 21643 ts_manager.cc:194] Registered new tserver with Master: 26227ae0411048fe84b8ca66d37eab65 (127.10.12.131:46595)
I20260924 03:23:41.944363 21643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.131@KRBTEST.COM'} at 127.10.12.131:34065
I20260924 03:23:41.949787 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:23:41.969866 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:23:41.954513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52768 (local address 127.10.12.190:44627)
0924 03:23:41.954679 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 03:23:41.954684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:23:41.955036 (+ 352us) server_negotiation.cc:407] Connection header received
0924 03:23:41.955185 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:23:41.955189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:23:41.955266 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:23:41.955357 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:23:41.956950 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:41.957732 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:41.959747 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:41.960285 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:41.960399 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:23:41.963142 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:23:41.963207 (+ 65us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:23:41.963212 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:23:41.963273 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:23:41.966982 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:23:41.967856 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:23:41.967861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:23:41.967864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:23:41.968066 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:23:41.968482 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:23:41.968486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:23:41.968489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:23:41.968796 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:23:41.968999 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 03:23:41.969415 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 03:23:41.969615 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:23:41.974996 21643 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:23:41.983796 21643 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:23:42.016383 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:23:41.997826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:41639 (local address 127.0.0.1:34300)
0924 03:23:42.006440 (+ 8614us) negotiation.cc:107] Waiting for socket to connect
0924 03:23:42.006507 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 03:23:42.006752 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:23:42.007938 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:23:42.007969 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:23:42.008020 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:23:42.008502 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:23:42.008518 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.010931 (+ 2413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.010936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:23:42.013519 (+ 2583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:23:42.013533 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.015374 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.015380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:23:42.015538 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:23:42.015568 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 03:23:42.015655 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 03:23:42.015872 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8294,"mutex_wait_us":295,"spinlock_wait_cycles":9984,"thread_start_us":591,"threads_started":1}
I20260924 03:23:42.016656 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:23:41.998948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39406 (local address 127.10.12.131:46595)
0924 03:23:42.006021 (+ 7073us) server_negotiation.cc:206] Beginning negotiation
0924 03:23:42.006033 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:23:42.007254 (+ 1221us) server_negotiation.cc:407] Connection header received
0924 03:23:42.007367 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:23:42.007376 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:23:42.007677 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:23:42.007876 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:23:42.008825 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.010898 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.013858 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.014826 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.014949 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:23:42.015207 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 03:23:42.016137 (+ 930us) server_negotiation.cc:299] Negotiation successful
0924 03:23:42.016305 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6823,"thread_start_us":6634,"threads_started":1}
I20260924 03:23:42.017035 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:23:41.999678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42606 (local address 127.10.12.130:38327)
0924 03:23:42.007680 (+ 8002us) server_negotiation.cc:206] Beginning negotiation
0924 03:23:42.007688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:23:42.008386 (+ 698us) server_negotiation.cc:407] Connection header received
0924 03:23:42.008484 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:23:42.008491 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:23:42.008731 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:23:42.008946 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:23:42.009813 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.010807 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.013889 (+ 3082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.015241 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.015368 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:23:42.015521 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 03:23:42.016477 (+ 956us) server_negotiation.cc:299] Negotiation successful
0924 03:23:42.016642 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7866,"thread_start_us":7751,"threads_started":1}
I20260924 03:23:42.017145 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:23:41.998512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34300 (local address 127.10.12.129:41639)
0924 03:23:42.007324 (+ 8812us) server_negotiation.cc:206] Beginning negotiation
0924 03:23:42.007332 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:23:42.007355 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:23:42.007449 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:23:42.007455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:23:42.007666 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:23:42.008750 (+ 1084us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:23:42.009050 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.010740 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.013698 (+ 2958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.015117 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.015318 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:23:42.015556 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 03:23:42.015803 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 03:23:42.016016 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8634,"thread_start_us":8446,"threads_started":1}
I20260924 03:23:42.017323 22146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:23:41.998396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:46595 (local address 127.0.0.1:39406)
0924 03:23:42.006440 (+ 8044us) negotiation.cc:107] Waiting for socket to connect
0924 03:23:42.006580 (+ 140us) client_negotiation.cc:174] Beginning negotiation
0924 03:23:42.006752 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:23:42.008190 (+ 1438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:23:42.008194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:23:42.008226 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:23:42.008641 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:23:42.008652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.011018 (+ 2366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.011027 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:23:42.013654 (+ 2627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:23:42.013671 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.015027 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.015034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:23:42.015227 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:23:42.015283 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 03:23:42.015424 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 03:23:42.016968 (+ 1544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7798,"mutex_wait_us":37,"spinlock_wait_cycles":6016,"thread_start_us":6462,"threads_started":1}
I20260924 03:23:42.017547 22150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:23:41.999422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:38327 (local address 127.0.0.1:42606)
0924 03:23:42.008055 (+ 8633us) negotiation.cc:107] Waiting for socket to connect
0924 03:23:42.008076 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:23:42.008243 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:23:42.009319 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:23:42.009327 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:23:42.009353 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:23:42.009641 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:23:42.009650 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.010974 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.010978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:23:42.013741 (+ 2763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:23:42.013752 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.015666 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.015673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:23:42.015858 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:23:42.015891 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 03:23:42.016053 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 03:23:42.016198 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8488,"mutex_wait_us":277,"spinlock_wait_cycles":768,"thread_start_us":8771,"threads_started":1}
I20260924 03:23:42.021242 22068 tablet_service.cc:1467] Processing CreateTablet for tablet 45e051dfb48c4a36af016212dc41fafb (DEFAULT_TABLE table=test-table [id=7bfb3925788d44b599cce0a2686a0420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:23:42.021242 21791 tablet_service.cc:1467] Processing CreateTablet for tablet 45e051dfb48c4a36af016212dc41fafb (DEFAULT_TABLE table=test-table [id=7bfb3925788d44b599cce0a2686a0420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:23:42.021243 21934 tablet_service.cc:1467] Processing CreateTablet for tablet 45e051dfb48c4a36af016212dc41fafb (DEFAULT_TABLE table=test-table [id=7bfb3925788d44b599cce0a2686a0420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:23:42.022590 21934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e051dfb48c4a36af016212dc41fafb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:23:42.022751 22068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e051dfb48c4a36af016212dc41fafb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:23:42.022769 21791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e051dfb48c4a36af016212dc41fafb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:23:42.035583 22152 tablet_bootstrap.cc:492] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658: Bootstrap starting.
I20260924 03:23:42.037850 22153 tablet_bootstrap.cc:492] T 45e051dfb48c4a36af016212dc41fafb P 0403757537c44724a898dd47d7f552a2: Bootstrap starting.
I20260924 03:23:42.038743 22154 tablet_bootstrap.cc:492] T 45e051dfb48c4a36af016212dc41fafb P 26227ae0411048fe84b8ca66d37eab65: Bootstrap starting.
I20260924 03:23:42.041082 22152 tablet_bootstrap.cc:654] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658: Neither blocks nor log segments found. Creating new log.
I20260924 03:23:42.042428 22154 tablet_bootstrap.cc:654] T 45e051dfb48c4a36af016212dc41fafb P 26227ae0411048fe84b8ca66d37eab65: Neither blocks nor log segments found. Creating new log.
I20260924 03:23:42.042418 22152 log.cc:824] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658: Log is configured to *not* fsync() on all Append() calls
I20260924 03:23:42.042613 22153 tablet_bootstrap.cc:654] T 45e051dfb48c4a36af016212dc41fafb P 0403757537c44724a898dd47d7f552a2: Neither blocks nor log segments found. Creating new log.
I20260924 03:23:42.043566 22154 log.cc:824] T 45e051dfb48c4a36af016212dc41fafb P 26227ae0411048fe84b8ca66d37eab65: Log is configured to *not* fsync() on all Append() calls
I20260924 03:23:42.043663 22153 log.cc:824] T 45e051dfb48c4a36af016212dc41fafb P 0403757537c44724a898dd47d7f552a2: Log is configured to *not* fsync() on all Append() calls
I20260924 03:23:42.045861 22152 tablet_bootstrap.cc:492] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658: No bootstrap required, opened a new log
I20260924 03:23:42.046132 22152 ts_tablet_manager.cc:1397] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260924 03:23:42.046715 22153 tablet_bootstrap.cc:492] T 45e051dfb48c4a36af016212dc41fafb P 0403757537c44724a898dd47d7f552a2: No bootstrap required, opened a new log
I20260924 03:23:42.047003 22153 ts_tablet_manager.cc:1397] T 45e051dfb48c4a36af016212dc41fafb P 0403757537c44724a898dd47d7f552a2: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 03:23:42.047037 22154 tablet_bootstrap.cc:492] T 45e051dfb48c4a36af016212dc41fafb P 26227ae0411048fe84b8ca66d37eab65: No bootstrap required, opened a new log
I20260924 03:23:42.047261 22154 ts_tablet_manager.cc:1397] T 45e051dfb48c4a36af016212dc41fafb P 26227ae0411048fe84b8ca66d37eab65: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260924 03:23:42.054733 22152 raft_consensus.cc:348] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0403757537c44724a898dd47d7f552a2" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 41639 } } peers { permanent_uuid: "d6558bed5c50498e9830889879192658" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 38327 } } peers { permanent_uuid: "26227ae0411048fe84b8ca66d37eab65" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46595 } }
I20260924 03:23:42.055153 22152 raft_consensus.cc:374] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:23:42.055266 22152 raft_consensus.cc:729] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6558bed5c50498e9830889879192658, State: Initialized, Role: FOLLOWER
I20260924 03:23:42.055871 22152 consensus_queue.cc:260] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0403757537c44724a898dd47d7f552a2" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 41639 } } peers { permanent_uuid: "d6558bed5c50498e9830889879192658" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 38327 } } peers { permanent_uuid: "26227ae0411048fe84b8ca66d37eab65" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46595 } }
I20260924 03:23:42.057060 21999 heartbeater.cc:503] Master 127.10.12.190:44627 was elected leader, sending a full tablet report...
I20260924 03:23:42.057809 22152 ts_tablet_manager.cc:1428] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260924 03:23:42.057616 22153 raft_consensus.cc:348] T 45e051dfb48c4a36af016212dc41fafb P 0403757537c44724a898dd47d7f552a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0403757537c44724a898dd47d7f552a2" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 41639 } } peers { permanent_uuid: "d6558bed5c50498e9830889879192658" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 38327 } } peers { permanent_uuid: "26227ae0411048fe84b8ca66d37eab65" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46595 } }
I20260924 03:23:42.057616 22154 raft_consensus.cc:348] T 45e051dfb48c4a36af016212dc41fafb P 26227ae0411048fe84b8ca66d37eab65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0403757537c44724a898dd47d7f552a2" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 41639 } } peers { permanent_uuid: "d6558bed5c50498e9830889879192658" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 38327 } } peers { permanent_uuid: "26227ae0411048fe84b8ca66d37eab65" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46595 } }
I20260924 03:23:42.058046 22153 raft_consensus.cc:374] T 45e051dfb48c4a36af016212dc41fafb P 0403757537c44724a898dd47d7f552a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:23:42.058044 22154 raft_consensus.cc:374] T 45e051dfb48c4a36af016212dc41fafb P 26227ae0411048fe84b8ca66d37eab65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:23:42.058135 22154 raft_consensus.cc:729] T 45e051dfb48c4a36af016212dc41fafb P 26227ae0411048fe84b8ca66d37eab65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26227ae0411048fe84b8ca66d37eab65, State: Initialized, Role: FOLLOWER
I20260924 03:23:42.058146 22153 raft_consensus.cc:729] T 45e051dfb48c4a36af016212dc41fafb P 0403757537c44724a898dd47d7f552a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0403757537c44724a898dd47d7f552a2, State: Initialized, Role: FOLLOWER
I20260924 03:23:42.058820 22153 consensus_queue.cc:260] T 45e051dfb48c4a36af016212dc41fafb P 0403757537c44724a898dd47d7f552a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0403757537c44724a898dd47d7f552a2" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 41639 } } peers { permanent_uuid: "d6558bed5c50498e9830889879192658" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 38327 } } peers { permanent_uuid: "26227ae0411048fe84b8ca66d37eab65" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46595 } }
I20260924 03:23:42.059386 22154 consensus_queue.cc:260] T 45e051dfb48c4a36af016212dc41fafb P 26227ae0411048fe84b8ca66d37eab65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0403757537c44724a898dd47d7f552a2" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 41639 } } peers { permanent_uuid: "d6558bed5c50498e9830889879192658" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 38327 } } peers { permanent_uuid: "26227ae0411048fe84b8ca66d37eab65" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46595 } }
I20260924 03:23:42.060689 22133 heartbeater.cc:503] Master 127.10.12.190:44627 was elected leader, sending a full tablet report...
I20260924 03:23:42.061467 22154 ts_tablet_manager.cc:1428] T 45e051dfb48c4a36af016212dc41fafb P 26227ae0411048fe84b8ca66d37eab65: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260924 03:23:42.063295 22153 ts_tablet_manager.cc:1428] T 45e051dfb48c4a36af016212dc41fafb P 0403757537c44724a898dd47d7f552a2: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
W20260924 03:23:42.104687 22000 tablet.cc:2369] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:23:42.163321 22158 raft_consensus.cc:482] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:23:42.163645 22158 raft_consensus.cc:504] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0403757537c44724a898dd47d7f552a2" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 41639 } } peers { permanent_uuid: "d6558bed5c50498e9830889879192658" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 38327 } } peers { permanent_uuid: "26227ae0411048fe84b8ca66d37eab65" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46595 } }
I20260924 03:23:42.165649 22158 leader_election.cc:288] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0403757537c44724a898dd47d7f552a2 (127.10.12.129:41639), 26227ae0411048fe84b8ca66d37eab65 (127.10.12.131:46595)
W20260924 03:23:42.168430 22134 tablet.cc:2369] T 45e051dfb48c4a36af016212dc41fafb P 26227ae0411048fe84b8ca66d37eab65: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:23:42.174971 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:23:42.166032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:44797 (local address 127.10.12.129:41639)
0924 03:23:42.166264 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 03:23:42.166269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:23:42.167259 (+ 990us) server_negotiation.cc:407] Connection header received
0924 03:23:42.167452 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:23:42.167457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:23:42.167531 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:23:42.167625 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:23:42.168585 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.169730 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.172535 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.173749 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.173904 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:23:42.173982 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 03:23:42.174563 (+ 581us) server_negotiation.cc:299] Negotiation successful
0924 03:23:42.174710 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260924 03:23:42.174971 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:23:42.166068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:34675 (local address 127.10.12.131:46595)
0924 03:23:42.166247 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 03:23:42.166253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:23:42.167261 (+ 1008us) server_negotiation.cc:407] Connection header received
0924 03:23:42.167452 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:23:42.167457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:23:42.167531 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:23:42.167624 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:23:42.168597 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.169420 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.172407 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.173701 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.173904 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:23:42.173982 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 03:23:42.174563 (+ 581us) server_negotiation.cc:299] Negotiation successful
0924 03:23:42.174710 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 03:23:42.175220 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:23:42.165994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:46595 (local address 127.10.12.130:34675)
0924 03:23:42.167174 (+ 1180us) negotiation.cc:107] Waiting for socket to connect
0924 03:23:42.167189 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:23:42.167336 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:23:42.167646 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:23:42.167651 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:23:42.167684 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:23:42.168432 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:23:42.168441 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.169565 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.169570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:23:42.172186 (+ 2616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:23:42.172206 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.174046 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.174053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:23:42.174293 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:23:42.174362 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 03:23:42.174475 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 03:23:42.174694 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1082,"mutex_wait_us":162,"spinlock_wait_cycles":15104,"thread_start_us":710,"threads_started":1}
I20260924 03:23:42.175239 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:23:42.165924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:41639 (local address 127.10.12.130:44797)
0924 03:23:42.167135 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0924 03:23:42.167158 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:23:42.167328 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:23:42.167646 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:23:42.167651 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:23:42.167687 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:23:42.168407 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:23:42.168425 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.169863 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.169868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:23:42.172351 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:23:42.172362 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.174047 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.174053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:23:42.174295 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:23:42.174374 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 03:23:42.174475 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 03:23:42.174694 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1057,"mutex_wait_us":157,"spinlock_wait_cycles":36736,"thread_start_us":868,"threads_started":1}
I20260924 03:23:42.176986 22088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e051dfb48c4a36af016212dc41fafb" candidate_uuid: "d6558bed5c50498e9830889879192658" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26227ae0411048fe84b8ca66d37eab65" is_pre_election: true
I20260924 03:23:42.176985 21811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e051dfb48c4a36af016212dc41fafb" candidate_uuid: "d6558bed5c50498e9830889879192658" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0403757537c44724a898dd47d7f552a2" is_pre_election: true
I20260924 03:23:42.177848 22088 raft_consensus.cc:2454] T 45e051dfb48c4a36af016212dc41fafb P 26227ae0411048fe84b8ca66d37eab65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6558bed5c50498e9830889879192658 in term 0.
I20260924 03:23:42.177848 21811 raft_consensus.cc:2454] T 45e051dfb48c4a36af016212dc41fafb P 0403757537c44724a898dd47d7f552a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6558bed5c50498e9830889879192658 in term 0.
I20260924 03:23:42.178683 21880 leader_election.cc:302] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0403757537c44724a898dd47d7f552a2, d6558bed5c50498e9830889879192658; no voters:
I20260924 03:23:42.179119 22158 raft_consensus.cc:2781] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:23:42.179299 22158 raft_consensus.cc:482] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:23:42.179418 22158 raft_consensus.cc:3037] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:23:42.181726 22158 raft_consensus.cc:504] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0403757537c44724a898dd47d7f552a2" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 41639 } } peers { permanent_uuid: "d6558bed5c50498e9830889879192658" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 38327 } } peers { permanent_uuid: "26227ae0411048fe84b8ca66d37eab65" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46595 } }
I20260924 03:23:42.182384 22158 leader_election.cc:288] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658 [CANDIDATE]: Term 1 election: Requested vote from peers 0403757537c44724a898dd47d7f552a2 (127.10.12.129:41639), 26227ae0411048fe84b8ca66d37eab65 (127.10.12.131:46595)
I20260924 03:23:42.183029 21811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e051dfb48c4a36af016212dc41fafb" candidate_uuid: "d6558bed5c50498e9830889879192658" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0403757537c44724a898dd47d7f552a2"
I20260924 03:23:42.183125 22088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e051dfb48c4a36af016212dc41fafb" candidate_uuid: "d6558bed5c50498e9830889879192658" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26227ae0411048fe84b8ca66d37eab65"
I20260924 03:23:42.183225 21811 raft_consensus.cc:3037] T 45e051dfb48c4a36af016212dc41fafb P 0403757537c44724a898dd47d7f552a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:23:42.183341 22088 raft_consensus.cc:3037] T 45e051dfb48c4a36af016212dc41fafb P 26227ae0411048fe84b8ca66d37eab65 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:23:42.185670 22088 raft_consensus.cc:2454] T 45e051dfb48c4a36af016212dc41fafb P 26227ae0411048fe84b8ca66d37eab65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6558bed5c50498e9830889879192658 in term 1.
I20260924 03:23:42.186225 21882 leader_election.cc:302] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26227ae0411048fe84b8ca66d37eab65, d6558bed5c50498e9830889879192658; no voters:
I20260924 03:23:42.186684 22158 raft_consensus.cc:2781] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:23:42.187424 22158 raft_consensus.cc:686] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658 [term 1 LEADER]: Becoming Leader. State: Replica: d6558bed5c50498e9830889879192658, State: Running, Role: LEADER
I20260924 03:23:42.187690 21811 raft_consensus.cc:2454] T 45e051dfb48c4a36af016212dc41fafb P 0403757537c44724a898dd47d7f552a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6558bed5c50498e9830889879192658 in term 1.
I20260924 03:23:42.188047 22158 consensus_queue.cc:237] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0403757537c44724a898dd47d7f552a2" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 41639 } } peers { permanent_uuid: "d6558bed5c50498e9830889879192658" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 38327 } } peers { permanent_uuid: "26227ae0411048fe84b8ca66d37eab65" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46595 } }
I20260924 03:23:42.193306 21642 catalog_manager.cc:5495] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6558bed5c50498e9830889879192658 (127.10.12.130). New cstate: current_term: 1 leader_uuid: "d6558bed5c50498e9830889879192658" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0403757537c44724a898dd47d7f552a2" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6558bed5c50498e9830889879192658" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 38327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26227ae0411048fe84b8ca66d37eab65" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46595 } health_report { overall_health: UNKNOWN } } }
I20260924 03:23:42.217077 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:23:42.211289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42620 (local address 127.10.12.130:38327)
0924 03:23:42.211492 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 03:23:42.211499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:23:42.211519 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:23:42.211723 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:23:42.211727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:23:42.211812 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:23:42.211944 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:23:42.212349 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.213140 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.214762 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.215186 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.215301 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:23:42.215821 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:23:42.216419 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:23:42.216557 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 03:23:42.216666 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 03:23:42.216758 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 03:23:42.226269 22088 raft_consensus.cc:1261] T 45e051dfb48c4a36af016212dc41fafb P 26227ae0411048fe84b8ca66d37eab65 [term 1 FOLLOWER]: Refusing update from remote peer d6558bed5c50498e9830889879192658: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:23:42.226270 21811 raft_consensus.cc:1261] T 45e051dfb48c4a36af016212dc41fafb P 0403757537c44724a898dd47d7f552a2 [term 1 FOLLOWER]: Refusing update from remote peer d6558bed5c50498e9830889879192658: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:23:42.227276 22163 consensus_queue.cc:1035] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26227ae0411048fe84b8ca66d37eab65" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 46595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:23:42.227720 22158 consensus_queue.cc:1035] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0403757537c44724a898dd47d7f552a2" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:23:42.240309 22166 mvcc.cc:205] Tried to move back new op lower bound from 7332742030228066304 to 7332742030086987776. Current Snapshot: MvccSnapshot[applied={T|T < 7332742030228066304}]
I20260924 03:23:42.241374 22168 mvcc.cc:205] Tried to move back new op lower bound from 7332742030228066304 to 7332742030086987776. Current Snapshot: MvccSnapshot[applied={T|T < 7332742030228066304}]
I20260924 03:23:42.255190 21642 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:23:42.255447 21642 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:23:42.259249 21642 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9086dd7395cd41da83f7d5bf11b21d22: Sending DeleteTablet for 3 replicas of tablet 45e051dfb48c4a36af016212dc41fafb
I20260924 03:23:42.260561 21791 tablet_service.cc:1514] Processing DeleteTablet for tablet 45e051dfb48c4a36af016212dc41fafb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:23:42 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:34300
I20260924 03:23:42.260589 22068 tablet_service.cc:1514] Processing DeleteTablet for tablet 45e051dfb48c4a36af016212dc41fafb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:23:42 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:39406
I20260924 03:23:42.263151 21934 tablet_service.cc:1514] Processing DeleteTablet for tablet 45e051dfb48c4a36af016212dc41fafb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:23:42 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:42606
I20260924 03:23:42.264063 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 21723
I20260924 03:23:42.273450 22176 tablet_replica.cc:330] T 45e051dfb48c4a36af016212dc41fafb P 26227ae0411048fe84b8ca66d37eab65: stopping tablet replica
I20260924 03:23:42.273981 22176 raft_consensus.cc:2229] T 45e051dfb48c4a36af016212dc41fafb P 26227ae0411048fe84b8ca66d37eab65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:23:42.274722 22176 raft_consensus.cc:2258] T 45e051dfb48c4a36af016212dc41fafb P 26227ae0411048fe84b8ca66d37eab65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:23:42.276037 22177 tablet_replica.cc:330] T 45e051dfb48c4a36af016212dc41fafb P 0403757537c44724a898dd47d7f552a2: stopping tablet replica
I20260924 03:23:42.276541 22177 raft_consensus.cc:2229] T 45e051dfb48c4a36af016212dc41fafb P 0403757537c44724a898dd47d7f552a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:23:42.277271 22177 raft_consensus.cc:2258] T 45e051dfb48c4a36af016212dc41fafb P 0403757537c44724a898dd47d7f552a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:23:42.280821 22177 ts_tablet_manager.cc:1905] T 45e051dfb48c4a36af016212dc41fafb P 0403757537c44724a898dd47d7f552a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:23:42.282172 22176 ts_tablet_manager.cc:1905] T 45e051dfb48c4a36af016212dc41fafb P 26227ae0411048fe84b8ca66d37eab65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:23:42.286144 22178 tablet_replica.cc:330] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658: stopping tablet replica
I20260924 03:23:42.290058 22178 raft_consensus.cc:2229] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:23:42.290246 22177 ts_tablet_manager.cc:1918] T 45e051dfb48c4a36af016212dc41fafb P 0403757537c44724a898dd47d7f552a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:23:42.290518 22177 log.cc:1196] T 45e051dfb48c4a36af016212dc41fafb P 0403757537c44724a898dd47d7f552a2: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-0/wal/wals/45e051dfb48c4a36af016212dc41fafb
I20260924 03:23:42.290987 22178 raft_consensus.cc:2258] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:23:42.291129 22177 ts_tablet_manager.cc:1939] T 45e051dfb48c4a36af016212dc41fafb P 0403757537c44724a898dd47d7f552a2: Deleting consensus metadata
I20260924 03:23:42.293735 21627 catalog_manager.cc:4841] TS 0403757537c44724a898dd47d7f552a2 (127.10.12.129:41639): tablet 45e051dfb48c4a36af016212dc41fafb (table test-table [id=7bfb3925788d44b599cce0a2686a0420]) successfully deleted
I20260924 03:23:42.295545 22176 ts_tablet_manager.cc:1918] T 45e051dfb48c4a36af016212dc41fafb P 26227ae0411048fe84b8ca66d37eab65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:23:42.295805 22176 log.cc:1196] T 45e051dfb48c4a36af016212dc41fafb P 26227ae0411048fe84b8ca66d37eab65: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/45e051dfb48c4a36af016212dc41fafb
I20260924 03:23:42.296389 22176 ts_tablet_manager.cc:1939] T 45e051dfb48c4a36af016212dc41fafb P 26227ae0411048fe84b8ca66d37eab65: Deleting consensus metadata
I20260924 03:23:42.298543 22178 ts_tablet_manager.cc:1905] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 03:23:42 dist-test-slave-00qs krb5kdc[21167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.129@KRBTEST.COM
I20260924 03:23:42.299042 21628 catalog_manager.cc:4841] TS 26227ae0411048fe84b8ca66d37eab65 (127.10.12.131:46595): tablet 45e051dfb48c4a36af016212dc41fafb (table test-table [id=7bfb3925788d44b599cce0a2686a0420]) successfully deleted
I20260924 03:23:42.305080 22178 ts_tablet_manager.cc:1918] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:23:42.305262 22178 log.cc:1196] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/45e051dfb48c4a36af016212dc41fafb
I20260924 03:23:42.305684 22178 ts_tablet_manager.cc:1939] T 45e051dfb48c4a36af016212dc41fafb P d6558bed5c50498e9830889879192658: Deleting consensus metadata
I20260924 03:23:42.307231 21627 catalog_manager.cc:4841] TS d6558bed5c50498e9830889879192658 (127.10.12.130:38327): tablet 45e051dfb48c4a36af016212dc41fafb (table test-table [id=7bfb3925788d44b599cce0a2686a0420]) successfully deleted
I20260924 03:23:42.308851 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:23:42.265659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34302 (local address 127.10.12.129:41639)
0924 03:23:42.265863 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 03:23:42.265871 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:23:42.282954 (+ 17083us) server_negotiation.cc:407] Connection header received
0924 03:23:42.283250 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:23:42.283257 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:23:42.283347 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:23:42.283437 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:23:42.286358 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.287161 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.291915 (+ 4754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.292830 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.292961 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:23:42.300637 (+ 7676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:23:42.300660 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:23:42.300682 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:23:42.300743 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:23:42.304873 (+ 4130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:23:42.305942 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:23:42.305953 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:23:42.305957 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:23:42.306132 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:23:42.306643 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:23:42.306648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:23:42.306652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:23:42.307168 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:23:42.307469 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 03:23:42.308175 (+ 706us) server_negotiation.cc:299] Negotiation successful
0924 03:23:42.308449 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:23:42.309633 21851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:23:42.454464 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 21723
I20260924 03:23:42.475165 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 21862
Sep 24 03:23:42 dist-test-slave-00qs krb5kdc[21167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.130@KRBTEST.COM
I20260924 03:23:42.493388 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:23:42.476004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42634 (local address 127.10.12.130:38327)
0924 03:23:42.476147 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 03:23:42.476151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:23:42.476204 (+ 53us) server_negotiation.cc:407] Connection header received
0924 03:23:42.476443 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:23:42.476447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:23:42.476534 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:23:42.476630 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:23:42.478534 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.479445 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.481041 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.481472 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.481596 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:23:42.486798 (+ 5202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:23:42.486819 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:23:42.486836 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:23:42.486890 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:23:42.490271 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:23:42.491130 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:23:42.491139 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:23:42.491141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:23:42.491311 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:23:42.491710 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:23:42.491714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:23:42.491716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:23:42.492165 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:23:42.492502 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0924 03:23:42.492807 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 03:23:42.493039 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 03:23:42.494141 21994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:23:42.620532 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 21862
I20260924 03:23:42.638310 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 22005
Sep 24 03:23:42 dist-test-slave-00qs krb5kdc[21167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.131@KRBTEST.COM
I20260924 03:23:42.659139 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:23:42.639317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39420 (local address 127.10.12.131:46595)
0924 03:23:42.639564 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 03:23:42.639569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:23:42.639590 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:23:42.639783 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:23:42.639789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:23:42.639882 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:23:42.640038 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:23:42.641958 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.643026 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.644876 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:23:42.645344 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:23:42.645467 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:23:42.651279 (+ 5812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:23:42.651303 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:23:42.651325 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:23:42.651381 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:23:42.655291 (+ 3910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:23:42.656570 (+ 1279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:23:42.656581 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:23:42.656584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:23:42.656773 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:23:42.657226 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:23:42.657231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:23:42.657233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:23:42.657723 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:23:42.658078 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0924 03:23:42.658444 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 03:23:42.658716 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260924 03:23:42.660012 22128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:23:42.760874 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 22005
I20260924 03:23:42.775893 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 21611
I20260924 03:23:42.777511 21673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:23:42.906929 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 21611
I20260924 03:23:42.926627 10290 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 03:23:43.264302 10290 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 03:23:43.264617 10290 mini_ranger.cc:66] Stopping Ranger...
I20260924 03:23:43.612512 10290 mini_ranger.cc:68] Stopped Ranger
2026-09-24 03:23:43.612 UTC [21197] LOG: received smart shutdown request
2026-09-24 03:23:43.617 UTC [21197] LOG: background worker "logical replication launcher" (PID 21206) exited with exit code 1
2026-09-24 03:23:43.617 UTC [21201] LOG: shutting down
2026-09-24 03:23:43.823 UTC [21197] LOG: database system is shut down
2026-09-24T03:23:43Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53709 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:23:44 dist-test-slave-00qs krb5kdc[22192](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:23:44 dist-test-slave-00qs krb5kdc[22192](info): set up 2 sockets
Sep 24 03:23:44 dist-test-slave-00qs krb5kdc[22192](info): commencing operation
krb5kdc: starting...
W20260924 03:23:45.016450 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.007s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:23:45 dist-test-slave-00qs krb5kdc[22192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:23:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:23:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.12.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.12.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.12.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/rangeradmin_127.10.12.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.12.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/rangeradmin_127.10.12.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.12.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.12.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.12.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/rangerlookup_127.10.12.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.12.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/rangerlookup_127.10.12.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.12.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.12.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.12.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/HTTP_127.10.12.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.12.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/HTTP_127.10.12.148@KRBTEST.COM.keytab.
I20260924 03:23:45.351635 10290 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskxoV79t/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/postgres -l logfile start
2026-09-24 03:23:48.307 UTC [22222] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 03:23:48.307 UTC [22222] LOG: listening on IPv4 address "127.10.12.148", port 41227
2026-09-24 03:23:48.312 UTC [22222] LOG: listening on Unix socket "/tmp/.s.PGSQL.41227"
2026-09-24 03:23:48.339 UTC [22225] LOG: database system was shut down at 2026-09-24 03:23:46 UTC
2026-09-24 03:23:48.343 UTC [22222] LOG: database system is ready to accept connections
I20260924 03:23:49.276427 10290 mini_postgres.cc:96] Postgres bound to 41227
2026-09-24 03:23:49.294 UTC [22233] FATAL: database "slave" does not exist
127.10.12.148:41227 - accepting connections
I20260924 03:23:49.295089 10290 mini_ranger.cc:161] Starting Ranger...
I20260924 03:23:49.322932 10290 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 03:23:49.954236 10290 mini_ranger.cc:87] Created ranger Postgres database
I20260924 03:23:49.954371 10290 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ranger-admin
2026-09-24 03:23:50,283 [I] DB FLAVOR :POSTGRES
2026-09-24 03:23:50,284 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 03:23:50,284 [I] Checking connection..
2026-09-24 03:23:50,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:41227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 03:23:50,575 [I] Checking connection passed.
2026-09-24 03:23:50,575 [I] --------- Verifying version history table ---------
2026-09-24 03:23:50,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:41227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:23:50,863 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 03:23:50,863 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:41227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:23:51,136 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 03:23:51,137 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 03:23:51,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:41227/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 03:23:51.394 UTC [22297] WARNING: there is no transaction in progress
2026-09-24 03:23:51,400 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 03:23:51,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:41227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:23:51,652 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 03:23:51,653 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 03:23:51,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:41227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 03:23:51,915 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:41227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 03:23:52,169 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:41227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-00qs.c.gcp-upstream.internal','N') ;"
2026-09-24 03:23:52,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:41227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 03:23:52,699 [I] Table x_portal_user does not exist in database ranger
2026-09-24 03:23:52,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:41227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 03:23:52,966 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 03:23:52,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:41227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 03:23:53,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:41227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 03:23:53,491 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 03:23:53,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:41227/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 03:23:54.584 UTC [22405] WARNING: there is no transaction in progress
2026-09-24 03:23:54.605 UTC [22405] WARNING: there is no transaction in progress
2026-09-24 03:23:54.625 UTC [22405] WARNING: there is no transaction in progress
2026-09-24 03:23:54.643 UTC [22405] WARNING: there is no transaction in progress
2026-09-24 03:23:54.661 UTC [22405] WARNING: there is no transaction in progress
2026-09-24 03:23:54.683 UTC [22405] WARNING: there is no transaction in progress
2026-09-24 03:23:54.747 UTC [22405] WARNING: there is no transaction in progress
2026-09-24 03:23:54.755 UTC [22405] WARNING: there is no transaction in progress
2026-09-24 03:23:54.774 UTC [22405] WARNING: there is no transaction in progress
2026-09-24 03:23:54.786 UTC [22405] WARNING: there is no transaction in progress
2026-09-24 03:23:54.797 UTC [22405] WARNING: there is no transaction in progress
2026-09-24 03:23:54.815 UTC [22405] WARNING: there is no transaction in progress
2026-09-24 03:23:54.826 UTC [22405] WARNING: there is no transaction in progress
2026-09-24 03:23:54.834 UTC [22405] WARNING: there is no transaction in progress
2026-09-24 03:23:54.843 UTC [22405] WARNING: there is no transaction in progress
2026-09-24 03:23:54.852 UTC [22405] WARNING: there is no transaction in progress
2026-09-24 03:23:55,648 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 03:23:55,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:41227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 03:23:55,897 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 03:23:55,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:41227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 03:23:56,170 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 03:23:56,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:41227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 03:23:56,433 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 03:23:56,433 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:41227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 03:23:56,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:41227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 03:23:56,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:41227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-00qs.c.gcp-upstream.internal';"
2026-09-24 03:23:57,213 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 03:23:57,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:41227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 03:23:57,482 [I] DB_PATCHES have already been applied
I20260924 03:23:57.489058 10290 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ranger-admin:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskxoV79t/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 03:23:57.489158 10290 mini_ranger.cc:193] Using host: 127.10.12.148
I20260924 03:23:57.503719 10290 mini_ranger.cc:227] Ranger admin URL: http://127.10.12.148:33599
Sep 24, 2026 3:23:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:23:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:23:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 3:23:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 3:23:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 3:23:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 3:23:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 03:23:58.538609 10290 mini_ranger.cc:160] Time spent starting Ranger: real 9.244s user 0.003s sys 0.041s
Sep 24, 2026 3:24:19 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 3:24:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 3:24:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 3:24:19 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 3:24:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 3:24:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 3:24:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 3:24:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 3:24:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 3:24:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 3:24:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 3:24:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 3:24:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 3:24:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 3:24:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 3:24:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 3:24:19 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 3:24:20 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 03:24:21.236274 10290 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.12.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.12.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.12.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/rangerkms_127.10.12.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.12.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/rangerkms_127.10.12.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.12.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.12.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.12.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/HTTP_127.10.12.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.12.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/HTTP_127.10.12.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 03:24:21 dist-test-slave-00qs krb5kdc[22192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220261, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 03:24:21.398204 10290 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 03:24:21.418262 10290 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 03:24:22.574805 10290 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 03:24:22.574955 10290 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ranger-kms
I20260924 03:24:22.574998 10290 mini_ranger_kms.cc:227] Using postgres at 127.10.12.148:41227
2026-09-24 03:24:23,004 [I] DB FLAVOR :POSTGRES
2026-09-24 03:24:23,004 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 03:24:23,005 [I] Checking connection
2026-09-24 03:24:23,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:41227/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 03:24:23,304 [I] connection success
2026-09-24 03:24:23,304 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 03:24:23,304 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 03:24:23,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:41227/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 03:24:23,599 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 03:24:23,600 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 03:24:23,600 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 03:24:23,600 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.12.148:41227/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 03:24:23,905 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 03:24:24.607299 10290 mini_ranger_kms.cc:310] Created kms service
I20260924 03:24:24.844980 10290 mini_ranger_kms.cc:326] Created kudu user
I20260924 03:24:24.897174 10290 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 03:24:24 dist-test-slave-00qs krb5kdc[22192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220261, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.12.148@KRBTEST.COM
I20260924 03:24:25.317299 10290 mini_ranger_kms.cc:384] Added ranger policy
I20260924 03:24:25.317615 10290 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ranger-kms:/tmp/dist-test-taskxoV79t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskxoV79t/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 03:24:25.317672 10290 mini_ranger_kms.cc:240] Using host: 127.10.12.148
I20260924 03:24:25.335536 10290 mini_ranger_kms.cc:276] Ranger KMS PID: 22586
I20260924 03:24:25.335680 10290 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.12.148:50989
Sep 24, 2026 3:24:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:24:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:24:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 3:24:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 3:24:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 3:24:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 03:24:27.394657 10290 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.996s user 0.072s sys 0.370s
I20260924 03:24:27.394896 10290 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 03:24:27.395056 10290 mini_ranger_kms.cc:401] 127.10.12.148:50989/kms/v1/keys
Sep 24 03:24:27 dist-test-slave-00qs krb5kdc[22192](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790220267, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.12.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 03:24:28 dist-test-slave-00qs krb5kdc[22192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220267, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.12.148@KRBTEST.COM for HTTP/127.10.12.148@KRBTEST.COM
Sep 24 03:24:28 dist-test-slave-00qs krb5kdc[22192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220267, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.12.148@KRBTEST.COM for HTTP/127.10.12.148@KRBTEST.COM
Sep 24 03:24:30 dist-test-slave-00qs krb5kdc[22192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:24:30.945276 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:42113
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:40609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:42113
--ranger_config_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.12.148:50989/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:24:31.360428 22636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:24:31.360812 22636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:24:31.360958 22636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:24:31.369007 22636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:24:31.369163 22636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:24:31.369242 22636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:24:31.369323 22636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:24:31.369388 22636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:24:31.381079 22636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:40609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.12.148:50989/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:42113
--ranger_config_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:42113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:24:31.382910 22636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:24:31.384608 22636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:24:31.398830 22641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:31.398986 22636 server_base.cc:1034] running on GCE node
W20260924 03:24:31.399165 22644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:31.399014 22642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:31.400637 22636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:24:31.402590 22636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:24:31.403883 22636 hybrid_clock.cc:648] HybridClock initialized: now 1790220271403839 us; error 56 us; skew 500 ppm
I20260924 03:24:31.404335 22636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:24:31 dist-test-slave-00qs krb5kdc[22192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220271, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:24:31.412541 22636 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:24:31.414698 22636 webserver.cc:460] Webserver started at http://127.10.12.190:38471/ using document root <none> and password file <none>
I20260924 03:24:31.415575 22636 fs_manager.cc:359] Metadata directory not provided
I20260924 03:24:31.415731 22636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:24:31.416208 22636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:24:31 dist-test-slave-00qs krb5kdc[22192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220271, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for HTTP/127.10.12.148@KRBTEST.COM
I20260924 03:24:31.482937 22636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "ccc0dd0fb4b34e6197372f964c44e124"
format_stamp: "Formatted at 2026-09-24 03:24:31 on dist-test-slave-00qs"
server_key: "c8d517109a30fa8a2fb8f446d240be8f"
server_key_iv: "473d9692c9d9f65609bb00b94a721adb"
server_key_version: "kuduclusterkey@0"
I20260924 03:24:31.484304 22636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "ccc0dd0fb4b34e6197372f964c44e124"
format_stamp: "Formatted at 2026-09-24 03:24:31 on dist-test-slave-00qs"
server_key: "c8d517109a30fa8a2fb8f446d240be8f"
server_key_iv: "473d9692c9d9f65609bb00b94a721adb"
server_key_version: "kuduclusterkey@0"
I20260924 03:24:31.493085 22636 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260924 03:24:31.515028 22651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:31.516961 22636 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:24:31.517176 22636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "ccc0dd0fb4b34e6197372f964c44e124"
format_stamp: "Formatted at 2026-09-24 03:24:31 on dist-test-slave-00qs"
server_key: "c8d517109a30fa8a2fb8f446d240be8f"
server_key_iv: "473d9692c9d9f65609bb00b94a721adb"
server_key_version: "kuduclusterkey@0"
I20260924 03:24:31.517475 22636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:24:31.545981 22636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:24:31.551132 22636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:24:31.551503 22636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:24:31.573169 22636 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:42113
I20260924 03:24:31.573160 22702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:42113 every 8 connection(s)
I20260924 03:24:31.574906 22636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:24:31.580631 22703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:24:31.583112 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 22636
I20260924 03:24:31.583438 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:24:31.584751 10290 external_mini_cluster.cc:1428] Setting key e2ff3d3ab01ad0a00592de6cf86a94a5
I20260924 03:24:31.599296 22703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124: Bootstrap starting.
I20260924 03:24:31.604718 22703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124: Neither blocks nor log segments found. Creating new log.
Sep 24 03:24:31 dist-test-slave-00qs krb5kdc[22192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:24:31.606345 22703 log.cc:824] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124: Log is configured to *not* fsync() on all Append() calls
I20260924 03:24:31.611260 22703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124: No bootstrap required, opened a new log
I20260924 03:24:31.617798 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:31.587944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35360 (local address 127.10.12.190:42113)
0924 03:24:31.589461 (+ 1517us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:31.589482 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:31.589550 (+ 68us) server_negotiation.cc:407] Connection header received
0924 03:24:31.592146 (+ 2596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:31.592192 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:31.592892 (+ 700us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:31.593497 (+ 605us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:31.596128 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:31.598289 (+ 2161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:31.600564 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:31.601589 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:31.601863 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:31.607419 (+ 5556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:31.607431 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:31.607458 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:31.607508 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:31.611733 (+ 4225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:31.612792 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:31.612811 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:31.612823 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:31.613072 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:31.613505 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:31.613511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:31.613514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:31.614448 (+ 934us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:31.614803 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:31.615352 (+ 549us) server_negotiation.cc:299] Negotiation successful
0924 03:24:31.616013 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1139,"thread_start_us":783,"threads_started":1}
I20260924 03:24:31.622143 22703 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc0dd0fb4b34e6197372f964c44e124" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 42113 } }
I20260924 03:24:31.622637 22703 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:24:31.622738 22703 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc0dd0fb4b34e6197372f964c44e124, State: Initialized, Role: FOLLOWER
I20260924 03:24:31.623416 22703 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc0dd0fb4b34e6197372f964c44e124" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 42113 } }
I20260924 03:24:31.623670 22703 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:24:31.623798 22703 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:24:31.624027 22703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:24:31.626502 22703 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc0dd0fb4b34e6197372f964c44e124" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 42113 } }
I20260924 03:24:31.627111 22703 leader_election.cc:302] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccc0dd0fb4b34e6197372f964c44e124; no voters:
I20260924 03:24:31.627831 22703 leader_election.cc:288] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:24:31.628088 22708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:24:31.629199 22708 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [term 1 LEADER]: Becoming Leader. State: Replica: ccc0dd0fb4b34e6197372f964c44e124, State: Running, Role: LEADER
I20260924 03:24:31.629805 22708 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc0dd0fb4b34e6197372f964c44e124" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 42113 } }
I20260924 03:24:31.630365 22703 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:24:31.633141 22710 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccc0dd0fb4b34e6197372f964c44e124. Latest consensus state: current_term: 1 leader_uuid: "ccc0dd0fb4b34e6197372f964c44e124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc0dd0fb4b34e6197372f964c44e124" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 42113 } } }
I20260924 03:24:31.633597 22710 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [sys.catalog]: This master's current role is: LEADER
I20260924 03:24:31.633868 22709 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccc0dd0fb4b34e6197372f964c44e124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc0dd0fb4b34e6197372f964c44e124" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 42113 } } }
I20260924 03:24:31.634186 22709 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [sys.catalog]: This master's current role is: LEADER
I20260924 03:24:31.635023 22716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:24:31.642005 22716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:24:31.643451 22703 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 03:24:31.670532 22716 catalog_manager.cc:1347] Generated new cluster ID: 9ad221febbe74aafa8b2226eee72f8f1
I20260924 03:24:31.670718 22716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:24:31.693729 22716 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:24:31.695503 22716 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:24:31.708644 22716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124: Generated new TSK 0
I20260924 03:24:31.709914 22716 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
I20260924 03:24:32.492697 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.12.148:50989/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:42113
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:40609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:24:32 dist-test-slave-00qs krb5kdc[22192](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790220272, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 03:24:32.974679 22748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:24:32.975122 22748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:24:32.975342 22748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:24:32.987615 22748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:24:32.987788 22748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:24:32.988071 22748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:24:33.005797 22748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:40609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.12.148:50989/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:42113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:24:33.007869 22748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:24:33.010322 22748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:24:33.029307 22759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:33.029780 22758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:33.041070 22761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:33.042622 22748 server_base.cc:1034] running on GCE node
I20260924 03:24:33.043489 22748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:24:33.049508 22748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:24:33.050788 22748 hybrid_clock.cc:648] HybridClock initialized: now 1790220273050714 us; error 72 us; skew 500 ppm
I20260924 03:24:33.051178 22748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:24:33 dist-test-slave-00qs krb5kdc[22192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220273, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:24:33.054657 22748 init.cc:377] Logged in from keytab as kudu/127.10.12.129@KRBTEST.COM (short username kudu)
I20260924 03:24:33.056391 22748 webserver.cc:460] Webserver started at http://127.10.12.129:39805/ using document root <none> and password file <none>
I20260924 03:24:33.057142 22748 fs_manager.cc:359] Metadata directory not provided
I20260924 03:24:33.057247 22748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:24:33.057543 22748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:24:33 dist-test-slave-00qs krb5kdc[22192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220273, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for HTTP/127.10.12.148@KRBTEST.COM
I20260924 03:24:33.076300 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "e8ad6acfe6a04d19a7830327e3d7b88a"
format_stamp: "Formatted at 2026-09-24 03:24:33 on dist-test-slave-00qs"
server_key: "40c13c768c6b7958d052f32b4149fd47"
server_key_iv: "91b3f38787bd42276e1a31f6d0a61163"
server_key_version: "kuduclusterkey@0"
I20260924 03:24:33.087108 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "e8ad6acfe6a04d19a7830327e3d7b88a"
format_stamp: "Formatted at 2026-09-24 03:24:33 on dist-test-slave-00qs"
server_key: "40c13c768c6b7958d052f32b4149fd47"
server_key_iv: "91b3f38787bd42276e1a31f6d0a61163"
server_key_version: "kuduclusterkey@0"
I20260924 03:24:33.099977 22748 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.011s sys 0.000s
I20260924 03:24:33.147451 22768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:33.149248 22748 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.009s sys 0.004s
I20260924 03:24:33.149554 22748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "e8ad6acfe6a04d19a7830327e3d7b88a"
format_stamp: "Formatted at 2026-09-24 03:24:33 on dist-test-slave-00qs"
server_key: "40c13c768c6b7958d052f32b4149fd47"
server_key_iv: "91b3f38787bd42276e1a31f6d0a61163"
server_key_version: "kuduclusterkey@0"
I20260924 03:24:33.149936 22748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:24:33.171559 22748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:24:33.176335 22748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:24:33.176767 22748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:24:33.177860 22748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:24:33.184288 22748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:24:33.184394 22748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:33.184564 22748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:24:33.184623 22748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:33.243690 22748 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:45007
I20260924 03:24:33.243711 22880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:45007 every 8 connection(s)
I20260924 03:24:33.245163 22748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:24:33.250602 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 22748
I20260924 03:24:33.250841 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:24:33.251688 10290 external_mini_cluster.cc:1428] Setting key 6aeb165ca6415372fa78d9016b63d76d
Sep 24 03:24:33 dist-test-slave-00qs krb5kdc[22192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220272, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for HTTP/127.10.12.148@KRBTEST.COM
Sep 24 03:24:33 dist-test-slave-00qs krb5kdc[22192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220273, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:24:33.312774 22884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:33.249853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:34457 (local address 127.10.12.190:42113)
0924 03:24:33.286776 (+ 36923us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:33.286784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:33.291255 (+ 4471us) server_negotiation.cc:407] Connection header received
0924 03:24:33.291352 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:33.291358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:33.291484 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:33.291598 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:33.295160 (+ 3562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:33.295980 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:33.298672 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:33.299307 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:33.299426 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:33.304241 (+ 4815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:33.304249 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:33.304254 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:33.304312 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:33.307989 (+ 3677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:33.309578 (+ 1589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:33.309584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:33.309587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:33.309791 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:33.310404 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:33.310409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:33.310413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:33.310841 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:33.310977 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:33.312107 (+ 1130us) server_negotiation.cc:299] Negotiation successful
0924 03:24:33.312366 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36753,"thread_start_us":36562,"threads_started":1}
I20260924 03:24:33.315165 22883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:33.249096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:42113 (local address 127.10.12.129:34457)
0924 03:24:33.287628 (+ 38532us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:33.287697 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:33.290637 (+ 2940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:33.293661 (+ 3024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:33.293674 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:33.294294 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:24:33.294930 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:33.294951 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:33.296171 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:33.296177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:33.298436 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:33.298458 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:33.299840 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:33.299848 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:33.300188 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:33.301025 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:24:33.301049 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:24:33.303858 (+ 2809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:24:33.308223 (+ 4365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:33.308233 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:33.308718 (+ 485us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:33.309403 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:33.309961 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:33.309966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:33.309969 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:33.310252 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:33.311081 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:24:33.311090 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:24:33.311608 (+ 518us) client_negotiation.cc:769] Sending connection context
0924 03:24:33.313554 (+ 1946us) client_negotiation.cc:240] Negotiation successful
0924 03:24:33.314110 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38020,"thread_start_us":42545,"threads_started":1}
I20260924 03:24:33.317061 22881 heartbeater.cc:347] Connected to a master server at 127.10.12.190:42113
I20260924 03:24:33.317380 22881 heartbeater.cc:464] Registering TS with master...
I20260924 03:24:33.318230 22881 heartbeater.cc:511] Master 127.10.12.190:42113 requested a full tablet report, sending...
I20260924 03:24:33.321034 22668 ts_manager.cc:194] Registered new tserver with Master: e8ad6acfe6a04d19a7830327e3d7b88a (127.10.12.129:45007)
I20260924 03:24:33.323186 22668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.129@KRBTEST.COM'} at 127.10.12.129:34457
WARNING: no policy specified for kudu/127.10.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
W20260924 03:24:33.375823 10290 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.10.12.130: real 0.121s user 0.000s sys 0.018s
I20260924 03:24:33.376878 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.12.148:50989/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:42113
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:40609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:24:33 dist-test-slave-00qs krb5kdc[22192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220272, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for HTTP/127.10.12.148@KRBTEST.COM
W20260924 03:24:33.845427 22887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:24:33.845940 22887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:24:33.846127 22887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:24:33.857743 22887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:24:33.857971 22887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:24:33.858178 22887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:24:33.872588 22887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:40609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.12.148:50989/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:42113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:24:33.873920 22887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:24:33.875224 22887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:24:33.892400 22895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:33.892843 22894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:33.898310 22887 server_base.cc:1034] running on GCE node
W20260924 03:24:33.898687 22897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:33.899466 22887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:24:33.900732 22887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:24:33.902020 22887 hybrid_clock.cc:648] HybridClock initialized: now 1790220273901943 us; error 59 us; skew 500 ppm
I20260924 03:24:33.902483 22887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:24:33 dist-test-slave-00qs krb5kdc[22192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220273, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:24:33.906922 22887 init.cc:377] Logged in from keytab as kudu/127.10.12.130@KRBTEST.COM (short username kudu)
I20260924 03:24:33.918524 22887 webserver.cc:460] Webserver started at http://127.10.12.130:37675/ using document root <none> and password file <none>
I20260924 03:24:33.919607 22887 fs_manager.cc:359] Metadata directory not provided
I20260924 03:24:33.919780 22887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:24:33.920240 22887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:24:33 dist-test-slave-00qs krb5kdc[22192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220273, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for HTTP/127.10.12.148@KRBTEST.COM
I20260924 03:24:33.942780 22887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "1afa5e95b5b744ad8dbcd69431122030"
format_stamp: "Formatted at 2026-09-24 03:24:33 on dist-test-slave-00qs"
server_key: "ef9fcdeb930f7b928ff289ba00a3a1ad"
server_key_iv: "aca021b910a801917a286241742c5c47"
server_key_version: "kuduclusterkey@0"
I20260924 03:24:33.944020 22887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "1afa5e95b5b744ad8dbcd69431122030"
format_stamp: "Formatted at 2026-09-24 03:24:33 on dist-test-slave-00qs"
server_key: "ef9fcdeb930f7b928ff289ba00a3a1ad"
server_key_iv: "aca021b910a801917a286241742c5c47"
server_key_version: "kuduclusterkey@0"
I20260924 03:24:33.951417 22887 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260924 03:24:33.972569 22911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:33.974334 22887 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260924 03:24:33.974634 22887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "1afa5e95b5b744ad8dbcd69431122030"
format_stamp: "Formatted at 2026-09-24 03:24:33 on dist-test-slave-00qs"
server_key: "ef9fcdeb930f7b928ff289ba00a3a1ad"
server_key_iv: "aca021b910a801917a286241742c5c47"
server_key_version: "kuduclusterkey@0"
I20260924 03:24:33.974989 22887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:24:33.999907 22887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:24:34.005342 22887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:24:34.005671 22887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:24:34.011473 22887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:24:34.016840 22887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:24:34.016940 22887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:34.017071 22887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:24:34.017127 22887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:34.059307 22887 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:45103
I20260924 03:24:34.059350 23023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:45103 every 8 connection(s)
I20260924 03:24:34.060989 22887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:24:34.062234 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 22887
I20260924 03:24:34.062404 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:24:34.062997 10290 external_mini_cluster.cc:1428] Setting key c5b5e7c1b92551b8a5d8a3902a898b87
Sep 24 03:24:34 dist-test-slave-00qs krb5kdc[22192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220273, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.131@KRBTEST.COM" created.
I20260924 03:24:34.133661 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:34.065745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:52315 (local address 127.10.12.190:42113)
0924 03:24:34.088083 (+ 22338us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:34.088091 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:34.092029 (+ 3938us) server_negotiation.cc:407] Connection header received
0924 03:24:34.112709 (+ 20680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:34.112716 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:34.112829 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:34.112912 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:34.116175 (+ 3263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:34.117289 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:34.119801 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:34.120504 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:34.120607 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:34.125447 (+ 4840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:34.125456 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:34.125461 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:34.125515 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:34.129309 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:34.130490 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:34.130497 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:34.130500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:34.130667 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:34.131309 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:34.131315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:34.131319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:34.131720 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:34.131858 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:34.132975 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0924 03:24:34.133248 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22151,"thread_start_us":26380,"threads_started":1}
I20260924 03:24:34.136289 23026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:34.064811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:42113 (local address 127.10.12.130:52315)
0924 03:24:34.089307 (+ 24496us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:34.089365 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:34.112199 (+ 22834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:34.113593 (+ 1394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:34.113607 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:34.114357 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:24:34.115302 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:34.115333 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:34.117540 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:34.117546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:34.119579 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:34.119609 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:34.120962 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:34.120969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:34.121270 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:34.122095 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:24:34.122126 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:24:34.125112 (+ 2986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:24:34.129542 (+ 4430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:34.129553 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:34.129575 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:34.130301 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:34.130855 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:34.130860 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:34.130864 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:34.131178 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:34.132024 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:24:34.132033 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:24:34.132522 (+ 489us) client_negotiation.cc:769] Sending connection context
0924 03:24:34.134244 (+ 1722us) client_negotiation.cc:240] Negotiation successful
0924 03:24:34.134845 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22426,"thread_start_us":23268,"threads_started":1}
I20260924 03:24:34.138541 23024 heartbeater.cc:347] Connected to a master server at 127.10.12.190:42113
I20260924 03:24:34.138942 23024 heartbeater.cc:464] Registering TS with master...
I20260924 03:24:34.139848 23024 heartbeater.cc:511] Master 127.10.12.190:42113 requested a full tablet report, sending...
I20260924 03:24:34.141857 22668 ts_manager.cc:194] Registered new tserver with Master: 1afa5e95b5b744ad8dbcd69431122030 (127.10.12.130:45103)
I20260924 03:24:34.142896 22668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.130@KRBTEST.COM'} at 127.10.12.130:52315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
I20260924 03:24:34.164947 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.12.148:50989/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:42113
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:40609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:24:34.328337 22881 heartbeater.cc:503] Master 127.10.12.190:42113 was elected leader, sending a full tablet report...
W20260924 03:24:34.552771 23030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:24:34.553071 23030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:24:34.553188 23030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:24:34.560961 23030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:24:34.561085 23030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:24:34.561225 23030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:24:34.571296 23030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:40609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.12.148:50989/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:42113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:24:34.572654 23030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:24:34.573987 23030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:24:34.587792 23035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:34.588222 23036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:34.588372 23038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:34.588850 23030 server_base.cc:1034] running on GCE node
I20260924 03:24:34.589501 23030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:24:34.590530 23030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:24:34.591771 23030 hybrid_clock.cc:648] HybridClock initialized: now 1790220274591711 us; error 52 us; skew 500 ppm
I20260924 03:24:34.592185 23030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:24:34 dist-test-slave-00qs krb5kdc[22192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220274, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:24:34.595525 23030 init.cc:377] Logged in from keytab as kudu/127.10.12.131@KRBTEST.COM (short username kudu)
I20260924 03:24:34.597204 23030 webserver.cc:460] Webserver started at http://127.10.12.131:45599/ using document root <none> and password file <none>
I20260924 03:24:34.597892 23030 fs_manager.cc:359] Metadata directory not provided
I20260924 03:24:34.598019 23030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:24:34.598320 23030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:24:34 dist-test-slave-00qs krb5kdc[22192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220274, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for HTTP/127.10.12.148@KRBTEST.COM
I20260924 03:24:34.614392 23030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "770d0400c9b44f57abe878743827d3e3"
format_stamp: "Formatted at 2026-09-24 03:24:34 on dist-test-slave-00qs"
server_key: "4ef1fb08dedcf0dbfec7232d064d463c"
server_key_iv: "4a1bc2ff79fa646b342246b8d1d09d65"
server_key_version: "kuduclusterkey@0"
I20260924 03:24:34.615356 23030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "770d0400c9b44f57abe878743827d3e3"
format_stamp: "Formatted at 2026-09-24 03:24:34 on dist-test-slave-00qs"
server_key: "4ef1fb08dedcf0dbfec7232d064d463c"
server_key_iv: "4a1bc2ff79fa646b342246b8d1d09d65"
server_key_version: "kuduclusterkey@0"
I20260924 03:24:34.621305 23030 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260924 03:24:34.636030 23045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:34.637528 23030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 03:24:34.637799 23030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "770d0400c9b44f57abe878743827d3e3"
format_stamp: "Formatted at 2026-09-24 03:24:34 on dist-test-slave-00qs"
server_key: "4ef1fb08dedcf0dbfec7232d064d463c"
server_key_iv: "4a1bc2ff79fa646b342246b8d1d09d65"
server_key_version: "kuduclusterkey@0"
I20260924 03:24:34.638063 23030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:24:34.654767 23030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:24:34.659205 23030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:24:34.659490 23030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:24:34.660473 23030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:24:34.661895 23030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:24:34.661974 23030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:34.662101 23030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:24:34.662153 23030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:34.694970 23030 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:37121
I20260924 03:24:34.695014 23157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:37121 every 8 connection(s)
I20260924 03:24:34.696276 23030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:24:34.699234 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 23030
I20260924 03:24:34.699426 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:24:34.700155 10290 external_mini_cluster.cc:1428] Setting key 64dbd122f4f6daf1d4ed09072c676c16
Sep 24 03:24:34 dist-test-slave-00qs krb5kdc[22192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220274, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:24:34.717410 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:34.698831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:41335 (local address 127.10.12.190:42113)
0924 03:24:34.699392 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:34.699398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:34.700534 (+ 1136us) server_negotiation.cc:407] Connection header received
0924 03:24:34.702666 (+ 2132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:34.702670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:34.702778 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:34.702969 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:34.704569 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:34.705302 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:34.706919 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:34.707363 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:34.707464 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:34.710937 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:34.710942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:34.710946 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:34.710983 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:34.713876 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:34.714857 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:34.714864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:34.714867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:34.715053 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:34.715647 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:34.715652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:34.715655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:34.716018 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:34.716204 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:34.716928 (+ 724us) server_negotiation.cc:299] Negotiation successful
0924 03:24:34.717147 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":234,"threads_started":1}
I20260924 03:24:34.718338 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:34.699515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:42113 (local address 127.10.12.131:41335)
0924 03:24:34.700349 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:34.700402 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:34.702330 (+ 1928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:34.703245 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:34.703258 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:34.703705 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:24:34.704360 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:34.704379 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:34.705433 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:34.705437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:34.706751 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:34.706765 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:34.707614 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:34.707619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:34.707828 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:34.708533 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:24:34.708559 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:24:34.710668 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:24:34.714102 (+ 3434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:34.714113 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:34.714133 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:34.714692 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:34.715219 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:34.715222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:34.715225 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:34.715523 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:34.716191 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:24:34.716200 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:24:34.716562 (+ 362us) client_negotiation.cc:769] Sending connection context
0924 03:24:34.716829 (+ 267us) client_negotiation.cc:240] Negotiation successful
0924 03:24:34.717262 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":346,"threads_started":1}
I20260924 03:24:34.719619 23158 heartbeater.cc:347] Connected to a master server at 127.10.12.190:42113
I20260924 03:24:34.719890 23158 heartbeater.cc:464] Registering TS with master...
I20260924 03:24:34.720582 23158 heartbeater.cc:511] Master 127.10.12.190:42113 requested a full tablet report, sending...
I20260924 03:24:34.722325 22668 ts_manager.cc:194] Registered new tserver with Master: 770d0400c9b44f57abe878743827d3e3 (127.10.12.131:37121)
I20260924 03:24:34.723168 22668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.131@KRBTEST.COM'} at 127.10.12.131:41335
I20260924 03:24:34.729363 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:24:34.731370 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 22748
Sep 24 03:24:34 dist-test-slave-00qs krb5kdc[22192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.129@KRBTEST.COM
I20260924 03:24:34.750902 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:34.732475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41410 (local address 127.10.12.129:45007)
0924 03:24:34.733050 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:34.733057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:34.733083 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:24:34.733149 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:34.733155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:34.733360 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:34.733506 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:34.735038 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:34.736696 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:34.738432 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:34.739157 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:34.739288 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:34.743908 (+ 4620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:34.743922 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:34.743937 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:34.744031 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:34.747377 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:34.748612 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:34.748621 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:34.748623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:34.748806 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:34.749233 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:34.749237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:34.749239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:34.749700 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:34.749955 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:34.750343 (+ 388us) server_negotiation.cc:299] Negotiation successful
0924 03:24:34.750604 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":280,"threads_started":1}
I20260924 03:24:34.752381 22876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:24:34.843070 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 22748
I20260924 03:24:34.858228 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 22887
Sep 24 03:24:34 dist-test-slave-00qs krb5kdc[22192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.130@KRBTEST.COM
I20260924 03:24:34.884490 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:34.859575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48252 (local address 127.10.12.130:45103)
0924 03:24:34.860367 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:34.860376 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:34.860409 (+ 33us) server_negotiation.cc:407] Connection header received
0924 03:24:34.860488 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:34.860499 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:34.860801 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:34.860957 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:34.862472 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:34.863718 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:34.865317 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:34.866070 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:34.866190 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:34.876241 (+ 10051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:34.876269 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:34.876294 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:34.876364 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:34.880208 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:34.881429 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:34.881448 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:34.881451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:34.881726 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:34.882306 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:34.882314 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:34.882317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:34.882834 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:34.883130 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:34.883609 (+ 479us) server_negotiation.cc:299] Negotiation successful
0924 03:24:34.884055 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":479,"threads_started":1}
I20260924 03:24:34.886282 23019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:24:34.976002 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 22887
I20260924 03:24:34.992498 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 23030
Sep 24 03:24:35 dist-test-slave-00qs krb5kdc[22192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.131@KRBTEST.COM
I20260924 03:24:35.013957 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:34.994040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39048 (local address 127.10.12.131:37121)
0924 03:24:34.994888 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:34.994897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:34.994933 (+ 36us) server_negotiation.cc:407] Connection header received
0924 03:24:34.995009 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:34.995017 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:34.995301 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:34.995459 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:34.996988 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:34.998219 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:34.999739 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:35.000795 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:35.000908 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:35.006385 (+ 5477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:35.006409 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:35.006430 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:35.006492 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:35.010198 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:35.011332 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:35.011344 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:35.011347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:35.011532 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:35.012093 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:35.012099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:35.012101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:35.012602 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:35.012920 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:35.013296 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 03:24:35.013593 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":436,"threads_started":1}
I20260924 03:24:35.015626 23153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:24:35.105943 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 23030
I20260924 03:24:35.120101 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 22636
I20260924 03:24:35.121788 22698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:24:35.244329 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 22636
I20260924 03:24:35.265353 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:42113
--webserver_interface=127.10.12.190
--webserver_port=38471
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:40609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:42113
--ranger_config_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.12.148:50989/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.12.148:50989/kms with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:24:35.663483 23170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:24:35.663914 23170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:24:35.664076 23170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:24:35.673117 23170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:24:35.673292 23170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:24:35.673357 23170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:24:35.673446 23170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:24:35.673520 23170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:24:35.685787 23170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:40609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.12.148:50989/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:42113
--ranger_config_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:42113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=38471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:24:35.687731 23170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:24:35.689663 23170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:24:35.706568 23175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:35.706596 23178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:35.706648 23176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:35.710183 23170 server_base.cc:1034] running on GCE node
I20260924 03:24:35.711496 23170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:24:35.713811 23170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:24:35.715124 23170 hybrid_clock.cc:648] HybridClock initialized: now 1790220275715054 us; error 62 us; skew 500 ppm
I20260924 03:24:35.715613 23170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:24:35 dist-test-slave-00qs krb5kdc[22192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220275, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:24:35.720036 23170 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:24:35.722245 23170 webserver.cc:460] Webserver started at http://127.10.12.190:38471/ using document root <none> and password file <none>
I20260924 03:24:35.723224 23170 fs_manager.cc:359] Metadata directory not provided
I20260924 03:24:35.723394 23170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 03:24:35 dist-test-slave-00qs krb5kdc[22192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220275, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for HTTP/127.10.12.148@KRBTEST.COM
I20260924 03:24:35.800621 23170 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.006s sys 0.001s
I20260924 03:24:35.805287 23186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:35.807474 23170 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:24:35.807765 23170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "ccc0dd0fb4b34e6197372f964c44e124"
format_stamp: "Formatted at 2026-09-24 03:24:31 on dist-test-slave-00qs"
server_key: "c8d517109a30fa8a2fb8f446d240be8f"
server_key_iv: "473d9692c9d9f65609bb00b94a721adb"
server_key_version: "kuduclusterkey@0"
I20260924 03:24:35.809098 23170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:24:35.832469 23170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:24:35.837949 23170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:24:35.838290 23170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:24:35.860590 23170 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:42113
I20260924 03:24:35.860613 23237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:42113 every 8 connection(s)
I20260924 03:24:35.862582 23170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:24:35.867880 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 23170
I20260924 03:24:35.868923 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.12.148:50989/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:45007
--local_ip_for_outbound_sockets=127.10.12.129
--tserver_master_addrs=127.10.12.190:42113
--webserver_port=39805
--webserver_interface=127.10.12.129
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:40609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.12.148:50989/kms with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:24:35.871795 23238 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:24:35.876147 23238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124: Bootstrap starting.
I20260924 03:24:35.900714 23238 log.cc:824] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124: Log is configured to *not* fsync() on all Append() calls
I20260924 03:24:35.912159 23238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:24:35.912820 23238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124: Bootstrap complete.
I20260924 03:24:35.923028 23238 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc0dd0fb4b34e6197372f964c44e124" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 42113 } }
I20260924 03:24:35.924158 23238 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc0dd0fb4b34e6197372f964c44e124, State: Initialized, Role: FOLLOWER
I20260924 03:24:35.925118 23238 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc0dd0fb4b34e6197372f964c44e124" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 42113 } }
I20260924 03:24:35.925431 23238 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:24:35.925565 23238 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:24:35.925774 23238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:24:35.931010 23238 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc0dd0fb4b34e6197372f964c44e124" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 42113 } }
I20260924 03:24:35.931783 23238 leader_election.cc:302] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccc0dd0fb4b34e6197372f964c44e124; no voters:
I20260924 03:24:35.932642 23238 leader_election.cc:288] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 03:24:35.932834 23242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 03:24:35.934118 23242 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [term 2 LEADER]: Becoming Leader. State: Replica: ccc0dd0fb4b34e6197372f964c44e124, State: Running, Role: LEADER
I20260924 03:24:35.934862 23242 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc0dd0fb4b34e6197372f964c44e124" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 42113 } }
I20260924 03:24:35.935422 23238 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:24:35.937997 23244 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccc0dd0fb4b34e6197372f964c44e124. Latest consensus state: current_term: 2 leader_uuid: "ccc0dd0fb4b34e6197372f964c44e124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc0dd0fb4b34e6197372f964c44e124" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 42113 } } }
I20260924 03:24:35.938647 23244 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [sys.catalog]: This master's current role is: LEADER
I20260924 03:24:35.939338 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ccc0dd0fb4b34e6197372f964c44e124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccc0dd0fb4b34e6197372f964c44e124" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 42113 } } }
I20260924 03:24:35.939772 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124 [sys.catalog]: This master's current role is: LEADER
I20260924 03:24:35.946738 23248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:24:35.958539 23248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:24:35.959908 23238 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 03:24:35.960397 23248 catalog_manager.cc:1259] Loaded cluster ID: 9ad221febbe74aafa8b2226eee72f8f1
I20260924 03:24:35.960649 23248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:24:35.966233 23248 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:24:35.969332 23248 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124: Loaded TSK: 0
I20260924 03:24:35.972460 23248 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 03:24:36.438019 23240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:24:36.438614 23240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:24:36.438794 23240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:24:36.452827 23240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:24:36.453049 23240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:24:36.453276 23240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:24:36.472117 23240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:40609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.12.148:50989/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:45007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=39805
--tserver_master_addrs=127.10.12.190:42113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:24:36.474575 23240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:24:36.476969 23240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:24:36.523761 23277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:36.528393 23276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:36.529086 23280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:36.533337 23240 server_base.cc:1034] running on GCE node
I20260924 03:24:36.534904 23240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:24:36.548445 23240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:24:36.554477 23240 hybrid_clock.cc:648] HybridClock initialized: now 1790220276554373 us; error 93 us; skew 500 ppm
I20260924 03:24:36.555294 23240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:24:36 dist-test-slave-00qs krb5kdc[22192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220276, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:24:36.567873 23240 init.cc:377] Logged in from keytab as kudu/127.10.12.129@KRBTEST.COM (short username kudu)
I20260924 03:24:36.585216 23240 webserver.cc:460] Webserver started at http://127.10.12.129:39805/ using document root <none> and password file <none>
I20260924 03:24:36.586858 23240 fs_manager.cc:359] Metadata directory not provided
I20260924 03:24:36.587144 23240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 03:24:36 dist-test-slave-00qs krb5kdc[22192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220276, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for HTTP/127.10.12.148@KRBTEST.COM
I20260924 03:24:36.629347 23240 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.001s sys 0.004s
I20260924 03:24:36.633209 23288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:36.635367 23240 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:24:36.635589 23240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "e8ad6acfe6a04d19a7830327e3d7b88a"
format_stamp: "Formatted at 2026-09-24 03:24:33 on dist-test-slave-00qs"
server_key: "40c13c768c6b7958d052f32b4149fd47"
server_key_iv: "91b3f38787bd42276e1a31f6d0a61163"
server_key_version: "kuduclusterkey@0"
I20260924 03:24:36.636901 23240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:24:36.670203 23240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:24:36.677838 23240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:24:36.678365 23240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:24:36.680155 23240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:24:36.682603 23240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:24:36.682747 23240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:36.682884 23240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:24:36.683004 23240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260924 03:24:36.742561 23240 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:45007
I20260924 03:24:36.742592 23400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:45007 every 8 connection(s)
I20260924 03:24:36.745065 23240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:24:36.747203 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 23240
I20260924 03:24:36.748196 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.12.148:50989/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:45103
--local_ip_for_outbound_sockets=127.10.12.130
--tserver_master_addrs=127.10.12.190:42113
--webserver_port=37675
--webserver_interface=127.10.12.130
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:40609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.12.148:50989/kms with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:24:36 dist-test-slave-00qs krb5kdc[22192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220276, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:24:36.798520 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:36.754576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:42113 (local address 127.10.12.129:46315)
0924 03:24:36.759307 (+ 4731us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:36.759372 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:36.762897 (+ 3525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:36.767720 (+ 4823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:36.767739 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:36.768629 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:24:36.769804 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:36.769879 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:36.773880 (+ 4001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:36.773894 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:36.776528 (+ 2634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:36.776565 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:36.778716 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:36.778727 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:36.779094 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:36.780264 (+ 1170us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:24:36.780305 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:24:36.783680 (+ 3375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:24:36.791114 (+ 7434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:36.791131 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:36.791156 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:36.792131 (+ 975us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:36.793040 (+ 909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:36.793047 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:36.793051 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:36.793433 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:36.794784 (+ 1351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:24:36.794796 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:24:36.795439 (+ 643us) client_negotiation.cc:769] Sending connection context
0924 03:24:36.795837 (+ 398us) client_negotiation.cc:240] Negotiation successful
0924 03:24:36.796716 (+ 879us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4302,"thread_start_us":3952,"threads_started":1}
I20260924 03:24:36.799479 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:36.757221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:46315 (local address 127.10.12.190:42113)
0924 03:24:36.764817 (+ 7596us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:36.764851 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:36.764937 (+ 86us) server_negotiation.cc:407] Connection header received
0924 03:24:36.765752 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:36.765788 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:36.766600 (+ 812us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:36.770447 (+ 3847us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:36.771310 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:36.773496 (+ 2186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:36.776916 (+ 3420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:36.778280 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:36.779815 (+ 1535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:36.784284 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:36.784300 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:36.784330 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:36.784402 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:36.790755 (+ 6353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:36.792480 (+ 1725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:36.792494 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:36.792506 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:36.792849 (+ 343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:36.793626 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:36.793632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:36.793636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:36.794569 (+ 933us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:36.796889 (+ 2320us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:36.797300 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 03:24:36.797932 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7136,"thread_start_us":12965,"threads_started":1}
I20260924 03:24:36.804167 23401 heartbeater.cc:347] Connected to a master server at 127.10.12.190:42113
I20260924 03:24:36.804622 23401 heartbeater.cc:464] Registering TS with master...
I20260924 03:24:36.805677 23401 heartbeater.cc:511] Master 127.10.12.190:42113 requested a full tablet report, sending...
W20260924 03:24:36.810647 23401 heartbeater.cc:644] Failed to heartbeat to 127.10.12.190:42113 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 03:24:36.811230 23401 heartbeater.cc:464] Registering TS with master...
I20260924 03:24:36.811631 23401 heartbeater.cc:511] Master 127.10.12.190:42113 requested a full tablet report, sending...
I20260924 03:24:36.813686 23401 heartbeater.cc:464] Registering TS with master...
I20260924 03:24:36.814237 23401 heartbeater.cc:511] Master 127.10.12.190:42113 requested a full tablet report, sending...
W20260924 03:24:36.816303 23401 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:24:37.251845 23405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:24:37.252333 23405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:24:37.252615 23405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:24:37.264976 23405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:24:37.265256 23405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:24:37.265491 23405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:24:37.281551 23405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:40609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.12.148:50989/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:45103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=37675
--tserver_master_addrs=127.10.12.190:42113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:24:37.283365 23405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:24:37.285431 23405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:24:37.307044 23421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:37.309484 23422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:37.319902 23405 server_base.cc:1034] running on GCE node
W20260924 03:24:37.320230 23425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:37.321200 23405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:24:37.322710 23405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:24:37.324050 23405 hybrid_clock.cc:648] HybridClock initialized: now 1790220277323975 us; error 66 us; skew 500 ppm
I20260924 03:24:37.324465 23405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:24:37 dist-test-slave-00qs krb5kdc[22192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220277, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:24:37.329245 23405 init.cc:377] Logged in from keytab as kudu/127.10.12.130@KRBTEST.COM (short username kudu)
I20260924 03:24:37.332368 23405 webserver.cc:460] Webserver started at http://127.10.12.130:37675/ using document root <none> and password file <none>
I20260924 03:24:37.333608 23405 fs_manager.cc:359] Metadata directory not provided
I20260924 03:24:37.333886 23405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 03:24:37 dist-test-slave-00qs krb5kdc[22192](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790220277, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 03:24:37 dist-test-slave-00qs krb5kdc[22192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220277, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for HTTP/127.10.12.148@KRBTEST.COM
I20260924 03:24:37.375394 23405 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.005s sys 0.004s
I20260924 03:24:37.386801 23432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:37.390050 23405 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.001s sys 0.009s
I20260924 03:24:37.390302 23405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "1afa5e95b5b744ad8dbcd69431122030"
format_stamp: "Formatted at 2026-09-24 03:24:33 on dist-test-slave-00qs"
server_key: "ef9fcdeb930f7b928ff289ba00a3a1ad"
server_key_iv: "aca021b910a801917a286241742c5c47"
server_key_version: "kuduclusterkey@0"
I20260924 03:24:37.392359 23405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:24:37.430015 23405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:24:37.438090 23405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:24:37.438545 23405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:24:37.440405 23405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:24:37.442461 23405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:24:37.442605 23405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:37.442731 23405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:24:37.442816 23405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:37.502336 23405 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:45103
I20260924 03:24:37.502409 23545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:45103 every 8 connection(s)
I20260924 03:24:37.504602 23405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:24:37.514892 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 23405
I20260924 03:24:37.515637 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.12.148:50989/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:37121
--local_ip_for_outbound_sockets=127.10.12.131
--tserver_master_addrs=127.10.12.190:42113
--webserver_port=45599
--webserver_interface=127.10.12.131
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:40609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.12.148:50989/kms with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:24:37 dist-test-slave-00qs krb5kdc[22192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220277, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:24:37.536602 23548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:37.508742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:55319 (local address 127.10.12.190:42113)
0924 03:24:37.509826 (+ 1084us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:37.509834 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:37.511116 (+ 1282us) server_negotiation.cc:407] Connection header received
0924 03:24:37.513987 (+ 2871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:37.513995 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:37.514101 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:37.514229 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:37.517106 (+ 2877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:37.518190 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:37.521551 (+ 3361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:37.522197 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:37.522366 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:37.527468 (+ 5102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:37.527475 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:37.527480 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:37.527531 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:37.531591 (+ 4060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:37.533099 (+ 1508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:37.533106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:37.533109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:37.533310 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:37.534067 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:37.534073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:37.534077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:37.534516 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:37.534720 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:37.535752 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0924 03:24:37.536087 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":931,"thread_start_us":813,"threads_started":1}
I20260924 03:24:37.538162 23549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:37.510003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:42113 (local address 127.10.12.130:55319)
0924 03:24:37.510885 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:37.510951 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:37.513514 (+ 2563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:37.514650 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:37.514665 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:37.515673 (+ 1008us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:24:37.516812 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:37.516847 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:37.518532 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:37.518543 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:37.521181 (+ 2638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:37.521208 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:37.522532 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:37.522542 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:37.522877 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:37.523829 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:24:37.523873 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:24:37.527077 (+ 3204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:24:37.531860 (+ 4783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:37.531872 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:37.531894 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:37.532888 (+ 994us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:37.533493 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:37.533498 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:37.533501 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:37.533879 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:37.534704 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:24:37.534715 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:24:37.535241 (+ 526us) client_negotiation.cc:769] Sending connection context
0924 03:24:37.535597 (+ 356us) client_negotiation.cc:240] Negotiation successful
0924 03:24:37.536272 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":326,"threads_started":1}
I20260924 03:24:37.541165 23546 heartbeater.cc:347] Connected to a master server at 127.10.12.190:42113
I20260924 03:24:37.541707 23546 heartbeater.cc:464] Registering TS with master...
I20260924 03:24:37.542768 23546 heartbeater.cc:511] Master 127.10.12.190:42113 requested a full tablet report, sending...
I20260924 03:24:37.545275 23203 ts_manager.cc:194] Registered new tserver with Master: 1afa5e95b5b744ad8dbcd69431122030 (127.10.12.130:45103)
I20260924 03:24:37.547639 23203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.130@KRBTEST.COM'} at 127.10.12.130:55319
I20260924 03:24:37.816453 23401 heartbeater.cc:464] Registering TS with master...
I20260924 03:24:37.817004 23401 heartbeater.cc:511] Master 127.10.12.190:42113 requested a full tablet report, sending...
I20260924 03:24:37.818665 23203 ts_manager.cc:194] Registered new tserver with Master: e8ad6acfe6a04d19a7830327e3d7b88a (127.10.12.129:45007)
I20260924 03:24:37.819897 23203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.129@KRBTEST.COM'} at 127.10.12.129:46315
Sep 24 03:24:37 dist-test-slave-00qs krb5kdc[22192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220277, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for HTTP/127.10.12.148@KRBTEST.COM
W20260924 03:24:38.007628 23550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:24:38.008143 23550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:24:38.008318 23550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:24:38.019789 23550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:24:38.020147 23550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:24:38.020377 23550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:24:38.036317 23550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:40609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.12.148:50989/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:37121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=45599
--tserver_master_addrs=127.10.12.190:42113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:24:38.038714 23550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:24:38.040695 23550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:24:38.058037 23556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:38.059054 23555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:38.059451 23558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:38.061981 23550 server_base.cc:1034] running on GCE node
I20260924 03:24:38.062860 23550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:24:38.064383 23550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:24:38.065721 23550 hybrid_clock.cc:648] HybridClock initialized: now 1790220278065663 us; error 54 us; skew 500 ppm
I20260924 03:24:38.066210 23550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:24:38 dist-test-slave-00qs krb5kdc[22192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220278, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:24:38.070912 23550 init.cc:377] Logged in from keytab as kudu/127.10.12.131@KRBTEST.COM (short username kudu)
I20260924 03:24:38.074124 23550 webserver.cc:460] Webserver started at http://127.10.12.131:45599/ using document root <none> and password file <none>
I20260924 03:24:38.075121 23550 fs_manager.cc:359] Metadata directory not provided
I20260924 03:24:38.075290 23550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 03:24:38 dist-test-slave-00qs krb5kdc[22192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220278, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for HTTP/127.10.12.148@KRBTEST.COM
I20260924 03:24:38.104934 23550 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 03:24:38.109206 23566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:38.111438 23550 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:24:38.111652 23550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "770d0400c9b44f57abe878743827d3e3"
format_stamp: "Formatted at 2026-09-24 03:24:34 on dist-test-slave-00qs"
server_key: "4ef1fb08dedcf0dbfec7232d064d463c"
server_key_iv: "4a1bc2ff79fa646b342246b8d1d09d65"
server_key_version: "kuduclusterkey@0"
I20260924 03:24:38.113111 23550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:24:38.141788 23550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:24:38.147430 23550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:24:38.147743 23550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:24:38.149291 23550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:24:38.151286 23550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:24:38.151408 23550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:38.151511 23550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:24:38.151590 23550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:24:38 dist-test-slave-00qs krb5kdc[22192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220277, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for HTTP/127.10.12.148@KRBTEST.COM
I20260924 03:24:38.277814 23550 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:37121
I20260924 03:24:38.277858 23679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:37121 every 8 connection(s)
I20260924 03:24:38.280005 23550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:24:38.283041 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 23550
Sep 24 03:24:38 dist-test-slave-00qs krb5kdc[22192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220278, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:24:38.345928 23684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:38.284814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:46979 (local address 127.10.12.190:42113)
0924 03:24:38.312961 (+ 28147us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:38.312969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:38.316027 (+ 3058us) server_negotiation.cc:407] Connection header received
0924 03:24:38.319413 (+ 3386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:38.319421 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:38.319538 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:38.319651 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:38.323316 (+ 3665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.324415 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.326662 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.327202 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.327318 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:38.335112 (+ 7794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:38.335120 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:38.335125 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:38.335193 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:38.339345 (+ 4152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:38.342042 (+ 2697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:38.342048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:38.342052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:38.342217 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:38.343418 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:38.343424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:38.343428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:38.343876 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:38.344056 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:38.345260 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0924 03:24:38.345522 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27964,"thread_start_us":32367,"threads_started":1}
I20260924 03:24:38.349104 23683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:38.284072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:42113 (local address 127.10.12.131:46979)
0924 03:24:38.313158 (+ 29086us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:38.313247 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:38.318916 (+ 5669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:38.320442 (+ 1526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:38.320458 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:38.321267 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:24:38.322348 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:38.322381 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.324613 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.324619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:38.326497 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:38.326512 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.327445 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.327452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:38.327677 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:38.331376 (+ 3699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:24:38.331412 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:24:38.334348 (+ 2936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:24:38.339912 (+ 5564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:38.339924 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:38.339946 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:38.340661 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:38.342468 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:38.342474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:38.342478 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:38.342807 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:38.344175 (+ 1368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:24:38.344184 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:24:38.344778 (+ 594us) client_negotiation.cc:769] Sending connection context
0924 03:24:38.346497 (+ 1719us) client_negotiation.cc:240] Negotiation successful
0924 03:24:38.347453 (+ 956us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28641,"thread_start_us":31892,"threads_started":1}
I20260924 03:24:38.351375 23680 heartbeater.cc:347] Connected to a master server at 127.10.12.190:42113
I20260924 03:24:38.351724 23680 heartbeater.cc:464] Registering TS with master...
I20260924 03:24:38.352687 23680 heartbeater.cc:511] Master 127.10.12.190:42113 requested a full tablet report, sending...
I20260924 03:24:38.354543 23203 ts_manager.cc:194] Registered new tserver with Master: 770d0400c9b44f57abe878743827d3e3 (127.10.12.131:37121)
I20260924 03:24:38.355713 23203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.131@KRBTEST.COM'} at 127.10.12.131:46979
I20260924 03:24:38.360522 23685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:38.317412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35366 (local address 127.10.12.190:42113)
0924 03:24:38.318765 (+ 1353us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:38.318771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:38.323594 (+ 4823us) server_negotiation.cc:407] Connection header received
0924 03:24:38.323792 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:38.323798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:38.323893 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:38.324067 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:38.326663 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.328157 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.330355 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.331114 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.331247 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:38.335585 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:38.335592 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:38.335597 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:38.335662 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:38.340702 (+ 5040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:38.341934 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:38.341940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:38.341944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:38.342158 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:38.352049 (+ 9891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:38.352056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:38.352061 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:38.352524 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:38.352864 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:38.359784 (+ 6920us) server_negotiation.cc:299] Negotiation successful
0924 03:24:38.360119 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":1203,"thread_start_us":1020,"threads_started":1}
I20260924 03:24:38.363718 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:24:38.396495 23685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:38.367515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35372 (local address 127.10.12.190:42113)
0924 03:24:38.367738 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:38.367744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:38.377589 (+ 9845us) server_negotiation.cc:407] Connection header received
0924 03:24:38.377758 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:38.377764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:38.377878 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:38.378006 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:38.380582 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.381742 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.383937 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.384637 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.384750 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:38.388745 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:38.388753 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:38.388758 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:38.388814 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:38.392735 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:38.393802 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:38.393809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:38.393813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:38.394007 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:38.394538 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:38.394543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:38.394547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:38.394930 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:38.395184 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:38.395827 (+ 643us) server_negotiation.cc:299] Negotiation successful
0924 03:24:38.396140 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 03:24:38.404744 23203 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:24:38.417351 23203 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:24:38.494871 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:38.459502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:45103 (local address 127.0.0.1:48258)
0924 03:24:38.478319 (+ 18817us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:38.478339 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:38.478482 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:38.479691 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:38.479698 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:38.479732 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:24:38.480364 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:38.480386 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.484069 (+ 3683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.484075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:38.491137 (+ 7062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:38.491155 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.493736 (+ 2581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.493743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:38.494001 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:38.494081 (+ 80us) client_negotiation.cc:769] Sending connection context
0924 03:24:38.494229 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 03:24:38.494461 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18684,"thread_start_us":16951,"threads_started":1}
I20260924 03:24:38.496061 23698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:38.460134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48258 (local address 127.10.12.130:45103)
0924 03:24:38.477887 (+ 17753us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:38.477896 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:38.478622 (+ 726us) server_negotiation.cc:407] Connection header received
0924 03:24:38.478740 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:38.478748 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:38.479027 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:38.479233 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:24:38.480620 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.483839 (+ 3219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.491373 (+ 7534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.493135 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.493289 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:38.493536 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:38.495468 (+ 1932us) server_negotiation.cc:299] Negotiation successful
0924 03:24:38.495691 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17567,"thread_start_us":17732,"threads_started":1}
I20260924 03:24:38.500105 23697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:38.457624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:37121 (local address 127.0.0.1:39062)
0924 03:24:38.476958 (+ 19334us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:38.477011 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:38.477279 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:38.488719 (+ 11440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:38.488725 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:38.488754 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:24:38.489090 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:38.489102 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.492889 (+ 3787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.492897 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:38.495090 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:38.495107 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.499130 (+ 4023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.499138 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:38.499322 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:38.499362 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 03:24:38.499545 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 03:24:38.499713 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18986,"thread_start_us":20441,"threads_started":1}
I20260924 03:24:38.501154 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:38.457938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39062 (local address 127.10.12.131:37121)
0924 03:24:38.487926 (+ 29988us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:38.487935 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:38.487993 (+ 58us) server_negotiation.cc:407] Connection header received
0924 03:24:38.488154 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:38.488164 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:38.488462 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:38.489288 (+ 826us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:24:38.489597 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.492702 (+ 3105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.495315 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.498374 (+ 3059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.500257 (+ 1883us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:38.500521 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:38.500607 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 03:24:38.500807 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29802,"thread_start_us":29643,"threads_started":1}
I20260924 03:24:38.503839 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:38.477699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:45007 (local address 127.0.0.1:41418)
0924 03:24:38.483299 (+ 5600us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:38.483323 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:38.483462 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:38.485237 (+ 1775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:38.485243 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:38.485274 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:24:38.487281 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:38.487298 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.496403 (+ 9105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.496410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:38.498539 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:38.498559 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.503004 (+ 4445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.503012 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:38.503216 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:38.503258 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 03:24:38.503337 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 03:24:38.503501 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5427,"thread_start_us":5847,"threads_started":1}
I20260924 03:24:38.505220 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:38.478023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41418 (local address 127.10.12.129:45007)
0924 03:24:38.484234 (+ 6211us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:38.484245 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:38.484279 (+ 34us) server_negotiation.cc:407] Connection header received
0924 03:24:38.484407 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:38.484416 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:38.484721 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:38.484972 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:24:38.487996 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.489695 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.498810 (+ 9115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.500459 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.504270 (+ 3811us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:38.504588 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:38.504680 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 03:24:38.504870 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6029,"thread_start_us":5813,"threads_started":1}
I20260924 03:24:38.505918 23481 tablet_service.cc:1467] Processing CreateTablet for tablet fecf9ec60d9f481d963b8a1cb46ffead (DEFAULT_TABLE table=test-table [id=d521d51682d84b2d83dc81ce105bbfcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:24:38.507606 23481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fecf9ec60d9f481d963b8a1cb46ffead. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:24:38.510253 23334 tablet_service.cc:1467] Processing CreateTablet for tablet fecf9ec60d9f481d963b8a1cb46ffead (DEFAULT_TABLE table=test-table [id=d521d51682d84b2d83dc81ce105bbfcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:24:38.512208 23334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fecf9ec60d9f481d963b8a1cb46ffead. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:24:38.518191 23614 tablet_service.cc:1467] Processing CreateTablet for tablet fecf9ec60d9f481d963b8a1cb46ffead (DEFAULT_TABLE table=test-table [id=d521d51682d84b2d83dc81ce105bbfcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:24:38.521649 23614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fecf9ec60d9f481d963b8a1cb46ffead. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:24:38.536835 23702 tablet_bootstrap.cc:492] T fecf9ec60d9f481d963b8a1cb46ffead P e8ad6acfe6a04d19a7830327e3d7b88a: Bootstrap starting.
I20260924 03:24:38.542027 23702 tablet_bootstrap.cc:654] T fecf9ec60d9f481d963b8a1cb46ffead P e8ad6acfe6a04d19a7830327e3d7b88a: Neither blocks nor log segments found. Creating new log.
I20260924 03:24:38.543707 23702 log.cc:824] T fecf9ec60d9f481d963b8a1cb46ffead P e8ad6acfe6a04d19a7830327e3d7b88a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:24:38.549017 23705 tablet_bootstrap.cc:492] T fecf9ec60d9f481d963b8a1cb46ffead P 770d0400c9b44f57abe878743827d3e3: Bootstrap starting.
I20260924 03:24:38.553968 23546 heartbeater.cc:503] Master 127.10.12.190:42113 was elected leader, sending a full tablet report...
I20260924 03:24:38.556767 23704 tablet_bootstrap.cc:492] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030: Bootstrap starting.
I20260924 03:24:38.559644 23705 tablet_bootstrap.cc:654] T fecf9ec60d9f481d963b8a1cb46ffead P 770d0400c9b44f57abe878743827d3e3: Neither blocks nor log segments found. Creating new log.
I20260924 03:24:38.560950 23704 tablet_bootstrap.cc:654] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030: Neither blocks nor log segments found. Creating new log.
I20260924 03:24:38.561045 23705 log.cc:824] T fecf9ec60d9f481d963b8a1cb46ffead P 770d0400c9b44f57abe878743827d3e3: Log is configured to *not* fsync() on all Append() calls
I20260924 03:24:38.562282 23704 log.cc:824] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030: Log is configured to *not* fsync() on all Append() calls
I20260924 03:24:38.563664 23702 tablet_bootstrap.cc:492] T fecf9ec60d9f481d963b8a1cb46ffead P e8ad6acfe6a04d19a7830327e3d7b88a: No bootstrap required, opened a new log
I20260924 03:24:38.564164 23702 ts_tablet_manager.cc:1397] T fecf9ec60d9f481d963b8a1cb46ffead P e8ad6acfe6a04d19a7830327e3d7b88a: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.013s
I20260924 03:24:38.568945 23704 tablet_bootstrap.cc:492] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030: No bootstrap required, opened a new log
I20260924 03:24:38.569272 23704 ts_tablet_manager.cc:1397] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20260924 03:24:38.577392 23702 raft_consensus.cc:348] T fecf9ec60d9f481d963b8a1cb46ffead P e8ad6acfe6a04d19a7830327e3d7b88a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d0400c9b44f57abe878743827d3e3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 37121 } } peers { permanent_uuid: "1afa5e95b5b744ad8dbcd69431122030" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45103 } } peers { permanent_uuid: "e8ad6acfe6a04d19a7830327e3d7b88a" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 45007 } }
I20260924 03:24:38.578100 23702 raft_consensus.cc:374] T fecf9ec60d9f481d963b8a1cb46ffead P e8ad6acfe6a04d19a7830327e3d7b88a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:24:38.578275 23702 raft_consensus.cc:729] T fecf9ec60d9f481d963b8a1cb46ffead P e8ad6acfe6a04d19a7830327e3d7b88a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8ad6acfe6a04d19a7830327e3d7b88a, State: Initialized, Role: FOLLOWER
I20260924 03:24:38.579124 23702 consensus_queue.cc:260] T fecf9ec60d9f481d963b8a1cb46ffead P e8ad6acfe6a04d19a7830327e3d7b88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d0400c9b44f57abe878743827d3e3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 37121 } } peers { permanent_uuid: "1afa5e95b5b744ad8dbcd69431122030" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45103 } } peers { permanent_uuid: "e8ad6acfe6a04d19a7830327e3d7b88a" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 45007 } }
I20260924 03:24:38.580569 23401 heartbeater.cc:503] Master 127.10.12.190:42113 was elected leader, sending a full tablet report...
I20260924 03:24:38.581416 23702 ts_tablet_manager.cc:1428] T fecf9ec60d9f481d963b8a1cb46ffead P e8ad6acfe6a04d19a7830327e3d7b88a: Time spent starting tablet: real 0.017s user 0.013s sys 0.001s
I20260924 03:24:38.582260 23705 tablet_bootstrap.cc:492] T fecf9ec60d9f481d963b8a1cb46ffead P 770d0400c9b44f57abe878743827d3e3: No bootstrap required, opened a new log
I20260924 03:24:38.582584 23705 ts_tablet_manager.cc:1397] T fecf9ec60d9f481d963b8a1cb46ffead P 770d0400c9b44f57abe878743827d3e3: Time spent bootstrapping tablet: real 0.035s user 0.014s sys 0.004s
I20260924 03:24:38.582300 23704 raft_consensus.cc:348] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d0400c9b44f57abe878743827d3e3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 37121 } } peers { permanent_uuid: "1afa5e95b5b744ad8dbcd69431122030" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45103 } } peers { permanent_uuid: "e8ad6acfe6a04d19a7830327e3d7b88a" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 45007 } }
I20260924 03:24:38.582970 23704 raft_consensus.cc:374] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:24:38.583123 23704 raft_consensus.cc:729] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1afa5e95b5b744ad8dbcd69431122030, State: Initialized, Role: FOLLOWER
I20260924 03:24:38.584024 23704 consensus_queue.cc:260] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d0400c9b44f57abe878743827d3e3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 37121 } } peers { permanent_uuid: "1afa5e95b5b744ad8dbcd69431122030" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45103 } } peers { permanent_uuid: "e8ad6acfe6a04d19a7830327e3d7b88a" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 45007 } }
I20260924 03:24:38.589717 23704 ts_tablet_manager.cc:1428] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030: Time spent starting tablet: real 0.020s user 0.012s sys 0.005s
I20260924 03:24:38.596343 23705 raft_consensus.cc:348] T fecf9ec60d9f481d963b8a1cb46ffead P 770d0400c9b44f57abe878743827d3e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d0400c9b44f57abe878743827d3e3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 37121 } } peers { permanent_uuid: "1afa5e95b5b744ad8dbcd69431122030" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45103 } } peers { permanent_uuid: "e8ad6acfe6a04d19a7830327e3d7b88a" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 45007 } }
I20260924 03:24:38.596891 23705 raft_consensus.cc:374] T fecf9ec60d9f481d963b8a1cb46ffead P 770d0400c9b44f57abe878743827d3e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:24:38.597007 23705 raft_consensus.cc:729] T fecf9ec60d9f481d963b8a1cb46ffead P 770d0400c9b44f57abe878743827d3e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770d0400c9b44f57abe878743827d3e3, State: Initialized, Role: FOLLOWER
I20260924 03:24:38.597705 23705 consensus_queue.cc:260] T fecf9ec60d9f481d963b8a1cb46ffead P 770d0400c9b44f57abe878743827d3e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d0400c9b44f57abe878743827d3e3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 37121 } } peers { permanent_uuid: "1afa5e95b5b744ad8dbcd69431122030" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45103 } } peers { permanent_uuid: "e8ad6acfe6a04d19a7830327e3d7b88a" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 45007 } }
I20260924 03:24:38.599044 23680 heartbeater.cc:503] Master 127.10.12.190:42113 was elected leader, sending a full tablet report...
I20260924 03:24:38.599706 23705 ts_tablet_manager.cc:1428] T fecf9ec60d9f481d963b8a1cb46ffead P 770d0400c9b44f57abe878743827d3e3: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20260924 03:24:38.631420 23715 raft_consensus.cc:482] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:24:38.631793 23715 raft_consensus.cc:504] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d0400c9b44f57abe878743827d3e3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 37121 } } peers { permanent_uuid: "1afa5e95b5b744ad8dbcd69431122030" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45103 } } peers { permanent_uuid: "e8ad6acfe6a04d19a7830327e3d7b88a" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 45007 } }
I20260924 03:24:38.634202 23715 leader_election.cc:288] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 770d0400c9b44f57abe878743827d3e3 (127.10.12.131:37121), e8ad6acfe6a04d19a7830327e3d7b88a (127.10.12.129:45007)
I20260924 03:24:38.642782 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:38.635626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:37121 (local address 127.10.12.130:39741)
0924 03:24:38.636542 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:38.636565 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:38.636710 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:38.637065 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:38.637070 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:38.637095 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:24:38.637472 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:38.637482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.638806 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.638814 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:38.640407 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:38.640426 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.641927 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.641935 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:38.642098 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:38.642135 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 03:24:38.642236 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 03:24:38.642384 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":584,"threads_started":1}
I20260924 03:24:38.642800 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:38.634751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:39741 (local address 127.10.12.131:37121)
0924 03:24:38.634968 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:38.634974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:38.636684 (+ 1710us) server_negotiation.cc:407] Connection header received
0924 03:24:38.636841 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:38.636846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:38.636930 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:38.637031 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:24:38.637668 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.638671 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.640656 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.641681 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.641856 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:38.641936 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:38.642298 (+ 362us) server_negotiation.cc:299] Negotiation successful
0924 03:24:38.642503 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 03:24:38.643500 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:38.634622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:36269 (local address 127.10.12.129:45007)
0924 03:24:38.634856 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:38.634863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:38.636101 (+ 1238us) server_negotiation.cc:407] Connection header received
0924 03:24:38.636452 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:38.636457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:38.636523 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:38.636581 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:24:38.637414 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.638447 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.641140 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.642294 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.642467 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:38.642538 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:38.643093 (+ 555us) server_negotiation.cc:299] Negotiation successful
0924 03:24:38.643256 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 03:24:38.644512 23717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:38.635126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:45007 (local address 127.10.12.130:36269)
0924 03:24:38.635991 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:38.636011 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:38.636237 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:38.636775 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:38.636780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:38.636858 (+ 78us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:24:38.637267 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:38.637278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.638606 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.638612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:38.640932 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:38.640951 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.642701 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.642708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:38.642886 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:38.642929 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 03:24:38.643691 (+ 762us) client_negotiation.cc:240] Negotiation successful
0924 03:24:38.644193 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":849,"threads_started":1}
I20260924 03:24:38.644359 23634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fecf9ec60d9f481d963b8a1cb46ffead" candidate_uuid: "1afa5e95b5b744ad8dbcd69431122030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "770d0400c9b44f57abe878743827d3e3" is_pre_election: true
I20260924 03:24:38.645205 23634 raft_consensus.cc:2454] T fecf9ec60d9f481d963b8a1cb46ffead P 770d0400c9b44f57abe878743827d3e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1afa5e95b5b744ad8dbcd69431122030 in term 0.
I20260924 03:24:38.646085 23433 leader_election.cc:302] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1afa5e95b5b744ad8dbcd69431122030, 770d0400c9b44f57abe878743827d3e3; no voters:
I20260924 03:24:38.645980 23356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fecf9ec60d9f481d963b8a1cb46ffead" candidate_uuid: "1afa5e95b5b744ad8dbcd69431122030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8ad6acfe6a04d19a7830327e3d7b88a" is_pre_election: true
I20260924 03:24:38.646585 23356 raft_consensus.cc:2454] T fecf9ec60d9f481d963b8a1cb46ffead P e8ad6acfe6a04d19a7830327e3d7b88a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1afa5e95b5b744ad8dbcd69431122030 in term 0.
I20260924 03:24:38.646548 23715 raft_consensus.cc:2781] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:24:38.646797 23715 raft_consensus.cc:482] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:24:38.646919 23715 raft_consensus.cc:3037] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:24:38.650354 23715 raft_consensus.cc:504] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d0400c9b44f57abe878743827d3e3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 37121 } } peers { permanent_uuid: "1afa5e95b5b744ad8dbcd69431122030" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45103 } } peers { permanent_uuid: "e8ad6acfe6a04d19a7830327e3d7b88a" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 45007 } }
I20260924 03:24:38.651239 23715 leader_election.cc:288] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030 [CANDIDATE]: Term 1 election: Requested vote from peers 770d0400c9b44f57abe878743827d3e3 (127.10.12.131:37121), e8ad6acfe6a04d19a7830327e3d7b88a (127.10.12.129:45007)
I20260924 03:24:38.651865 23634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fecf9ec60d9f481d963b8a1cb46ffead" candidate_uuid: "1afa5e95b5b744ad8dbcd69431122030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "770d0400c9b44f57abe878743827d3e3"
I20260924 03:24:38.652129 23634 raft_consensus.cc:3037] T fecf9ec60d9f481d963b8a1cb46ffead P 770d0400c9b44f57abe878743827d3e3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:24:38.652081 23356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fecf9ec60d9f481d963b8a1cb46ffead" candidate_uuid: "1afa5e95b5b744ad8dbcd69431122030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8ad6acfe6a04d19a7830327e3d7b88a"
I20260924 03:24:38.652304 23356 raft_consensus.cc:3037] T fecf9ec60d9f481d963b8a1cb46ffead P e8ad6acfe6a04d19a7830327e3d7b88a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:24:38.654609 23634 raft_consensus.cc:2454] T fecf9ec60d9f481d963b8a1cb46ffead P 770d0400c9b44f57abe878743827d3e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1afa5e95b5b744ad8dbcd69431122030 in term 1.
I20260924 03:24:38.655074 23433 leader_election.cc:302] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1afa5e95b5b744ad8dbcd69431122030, 770d0400c9b44f57abe878743827d3e3; no voters:
I20260924 03:24:38.655393 23715 raft_consensus.cc:2781] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:24:38.655541 23356 raft_consensus.cc:2454] T fecf9ec60d9f481d963b8a1cb46ffead P e8ad6acfe6a04d19a7830327e3d7b88a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1afa5e95b5b744ad8dbcd69431122030 in term 1.
I20260924 03:24:38.656076 23715 raft_consensus.cc:686] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030 [term 1 LEADER]: Becoming Leader. State: Replica: 1afa5e95b5b744ad8dbcd69431122030, State: Running, Role: LEADER
I20260924 03:24:38.656580 23715 consensus_queue.cc:237] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d0400c9b44f57abe878743827d3e3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 37121 } } peers { permanent_uuid: "1afa5e95b5b744ad8dbcd69431122030" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45103 } } peers { permanent_uuid: "e8ad6acfe6a04d19a7830327e3d7b88a" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 45007 } }
I20260924 03:24:38.662180 23203 catalog_manager.cc:5495] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1afa5e95b5b744ad8dbcd69431122030 (127.10.12.130). New cstate: current_term: 1 leader_uuid: "1afa5e95b5b744ad8dbcd69431122030" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770d0400c9b44f57abe878743827d3e3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 37121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1afa5e95b5b744ad8dbcd69431122030" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8ad6acfe6a04d19a7830327e3d7b88a" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 45007 } health_report { overall_health: UNKNOWN } } }
I20260924 03:24:38.697825 23698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:38.691304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48272 (local address 127.10.12.130:45103)
0924 03:24:38.691526 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:38.691532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:38.691561 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:24:38.691674 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:38.691680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:38.691789 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:38.691993 (+ 204us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:24:38.692400 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.693536 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.695009 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.695622 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.695830 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:38.696343 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:24:38.697113 (+ 770us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:24:38.697284 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:38.697435 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 03:24:38.697534 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 03:24:38.706856 23634 raft_consensus.cc:1261] T fecf9ec60d9f481d963b8a1cb46ffead P 770d0400c9b44f57abe878743827d3e3 [term 1 FOLLOWER]: Refusing update from remote peer 1afa5e95b5b744ad8dbcd69431122030: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:24:38.706949 23356 raft_consensus.cc:1261] T fecf9ec60d9f481d963b8a1cb46ffead P e8ad6acfe6a04d19a7830327e3d7b88a [term 1 FOLLOWER]: Refusing update from remote peer 1afa5e95b5b744ad8dbcd69431122030: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:24:38.707983 23715 consensus_queue.cc:1035] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ad6acfe6a04d19a7830327e3d7b88a" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 45007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:24:38.708357 23719 consensus_queue.cc:1035] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "770d0400c9b44f57abe878743827d3e3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 37121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:24:38.725281 23722 mvcc.cc:205] Tried to move back new op lower bound from 7332742261572460544 to 7332742261381435392. Current Snapshot: MvccSnapshot[applied={T|T < 7332742261572460544}]
I20260924 03:24:38.755204 23203 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:24:38.755551 23203 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:24:38.761083 23203 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ccc0dd0fb4b34e6197372f964c44e124: Sending DeleteTablet for 3 replicas of tablet fecf9ec60d9f481d963b8a1cb46ffead
I20260924 03:24:38.762384 23614 tablet_service.cc:1514] Processing DeleteTablet for tablet fecf9ec60d9f481d963b8a1cb46ffead with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:24:38 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:39062
I20260924 03:24:38.763929 23481 tablet_service.cc:1514] Processing DeleteTablet for tablet fecf9ec60d9f481d963b8a1cb46ffead with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:24:38 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:48258
I20260924 03:24:38.768187 23334 tablet_service.cc:1514] Processing DeleteTablet for tablet fecf9ec60d9f481d963b8a1cb46ffead with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:24:38 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:41418
I20260924 03:24:38.776600 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 23240
I20260924 03:24:38.776808 23732 tablet_replica.cc:330] T fecf9ec60d9f481d963b8a1cb46ffead P 770d0400c9b44f57abe878743827d3e3: stopping tablet replica
I20260924 03:24:38.777308 23732 raft_consensus.cc:2229] T fecf9ec60d9f481d963b8a1cb46ffead P 770d0400c9b44f57abe878743827d3e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:24:38.777832 23732 raft_consensus.cc:2258] T fecf9ec60d9f481d963b8a1cb46ffead P 770d0400c9b44f57abe878743827d3e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:24:38.778656 23733 tablet_replica.cc:330] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030: stopping tablet replica
I20260924 03:24:38.779064 23733 raft_consensus.cc:2229] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:24:38.779685 23733 raft_consensus.cc:2258] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:24:38.782020 23732 ts_tablet_manager.cc:1905] T fecf9ec60d9f481d963b8a1cb46ffead P 770d0400c9b44f57abe878743827d3e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:24:38.786816 23734 tablet_replica.cc:330] T fecf9ec60d9f481d963b8a1cb46ffead P e8ad6acfe6a04d19a7830327e3d7b88a: stopping tablet replica
I20260924 03:24:38.787307 23734 raft_consensus.cc:2229] T fecf9ec60d9f481d963b8a1cb46ffead P e8ad6acfe6a04d19a7830327e3d7b88a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:24:38.787899 23734 raft_consensus.cc:2258] T fecf9ec60d9f481d963b8a1cb46ffead P e8ad6acfe6a04d19a7830327e3d7b88a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:24:38.788668 23733 ts_tablet_manager.cc:1905] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:24:38.792380 23734 ts_tablet_manager.cc:1905] T fecf9ec60d9f481d963b8a1cb46ffead P e8ad6acfe6a04d19a7830327e3d7b88a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:24:38.796797 23733 ts_tablet_manager.cc:1918] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:24:38.797015 23733 log.cc:1196] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/fecf9ec60d9f481d963b8a1cb46ffead
I20260924 03:24:38.797586 23733 ts_tablet_manager.cc:1939] T fecf9ec60d9f481d963b8a1cb46ffead P 1afa5e95b5b744ad8dbcd69431122030: Deleting consensus metadata
I20260924 03:24:38.798182 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:38.778670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41430 (local address 127.10.12.129:45007)
0924 03:24:38.778850 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:38.778856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:38.778876 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:24:38.778942 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:38.778947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:38.779028 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:38.779123 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:38.781871 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.783162 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.785521 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.786324 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.786607 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:38.790109 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:38.790122 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:38.790139 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:38.790192 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:38.794486 (+ 4294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:38.795547 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:38.795557 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:38.795561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:38.795729 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:38.796176 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:38.796181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:38.796183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:38.796694 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:38.797196 (+ 502us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:38.797408 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 03:24:38.797822 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 03:24:38.799302 23396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:24:38.799831 23190 catalog_manager.cc:4841] TS 1afa5e95b5b744ad8dbcd69431122030 (127.10.12.130:45103): tablet fecf9ec60d9f481d963b8a1cb46ffead (table test-table [id=d521d51682d84b2d83dc81ce105bbfcc]) successfully deleted
I20260924 03:24:38.802910 23732 ts_tablet_manager.cc:1918] T fecf9ec60d9f481d963b8a1cb46ffead P 770d0400c9b44f57abe878743827d3e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:24:38.803108 23732 log.cc:1196] T fecf9ec60d9f481d963b8a1cb46ffead P 770d0400c9b44f57abe878743827d3e3: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/fecf9ec60d9f481d963b8a1cb46ffead
I20260924 03:24:38.803536 23732 ts_tablet_manager.cc:1939] T fecf9ec60d9f481d963b8a1cb46ffead P 770d0400c9b44f57abe878743827d3e3: Deleting consensus metadata
I20260924 03:24:38.805570 23187 catalog_manager.cc:4841] TS 770d0400c9b44f57abe878743827d3e3 (127.10.12.131:37121): tablet fecf9ec60d9f481d963b8a1cb46ffead (table test-table [id=d521d51682d84b2d83dc81ce105bbfcc]) successfully deleted
I20260924 03:24:38.959447 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 23240
I20260924 03:24:38.975329 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 23405
W20260924 03:24:38.975399 23190 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.12.129:45007 (error 108)
I20260924 03:24:38.977451 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:38.976929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:45007 (local address 127.0.0.1:41436)
0924 03:24:38.977135 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:38.977267 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 03:24:38.977761 23190 catalog_manager.cc:4584] TS e8ad6acfe6a04d19a7830327e3d7b88a (127.10.12.129:45007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fecf9ec60d9f481d963b8a1cb46ffead: Network error: Client connection negotiation failed: client connection to 127.10.12.129:45007: connect: Connection refused (error 111)
I20260924 03:24:38.994068 23698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:38.976535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48286 (local address 127.10.12.130:45103)
0924 03:24:38.976759 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:38.976765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:38.976788 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:24:38.977053 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:38.977059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:38.977139 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:38.977221 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:38.979128 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.980040 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.981968 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:38.982416 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:38.982529 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:38.986244 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:38.986270 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:38.986334 (+ 64us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:38.986421 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:38.990240 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:38.991366 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:38.991375 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:38.991378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:38.991560 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:38.992301 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:38.992305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:38.992308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:38.992763 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:38.993106 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:38.993412 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 03:24:38.993704 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260924 03:24:38.995023 23541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:24:39.031695 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:39.031138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:45007 (local address 127.0.0.1:41438)
0924 03:24:39.031389 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:39.031506 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260924 03:24:39.032859 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:39.032442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:45007 (local address 127.0.0.1:41446)
0924 03:24:39.032649 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:39.032725 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 03:24:39.093498 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:39.092945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:45007 (local address 127.0.0.1:41450)
0924 03:24:39.093203 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:39.093303 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260924 03:24:39.094552 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:39.094221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:45007 (local address 127.0.0.1:41454)
0924 03:24:39.094353 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:39.094440 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 03:24:39.111344 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 23405
I20260924 03:24:39.126338 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 23550
I20260924 03:24:39.140763 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:39.127173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.10.12.131:37121)
0924 03:24:39.127358 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:39.127364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:39.127424 (+ 60us) server_negotiation.cc:407] Connection header received
0924 03:24:39.127601 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:39.127604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:39.127771 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:39.127908 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:39.129344 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:39.130050 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:39.131713 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:39.132205 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:39.132324 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:39.134907 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:39.134923 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:39.134940 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:39.134982 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:39.137838 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:39.138625 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:39.138633 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:39.138636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:39.138776 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:39.139140 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:39.139144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:39.139146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:39.139526 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:39.139813 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:39.140201 (+ 388us) server_negotiation.cc:299] Negotiation successful
0924 03:24:39.140418 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:24:39.141542 23675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:24:39.175768 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:39.175114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:45007 (local address 127.0.0.1:41466)
0924 03:24:39.175389 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:39.175559 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260924 03:24:39.176964 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:39.176540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:45007 (local address 127.0.0.1:41470)
0924 03:24:39.176748 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:39.176830 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260924 03:24:39.231943 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 23550
I20260924 03:24:39.244939 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 23170
I20260924 03:24:39.246239 23233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:24:39.362135 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:39.360675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:45007 (local address 127.0.0.1:41484)
0924 03:24:39.361680 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:39.361821 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":771}
I20260924 03:24:39.362622 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 23170
I20260924 03:24:39.384454 10290 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 03:24:39.720772 10290 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 03:24:39.721012 10290 mini_ranger.cc:66] Stopping Ranger...
I20260924 03:24:40.068503 10290 mini_ranger.cc:68] Stopped Ranger
2026-09-24 03:24:40.068 UTC [22222] LOG: received smart shutdown request
2026-09-24 03:24:40.073 UTC [22222] LOG: background worker "logical replication launcher" (PID 22231) exited with exit code 1
2026-09-24 03:24:40.073 UTC [22226] LOG: shutting down
2026-09-24 03:24:40.322 UTC [22222] LOG: database system is shut down
2026-09-24T03:24:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (56509 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:24:40 dist-test-slave-00qs krb5kdc[23748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:24:40 dist-test-slave-00qs krb5kdc[23748](info): set up 2 sockets
Sep 24 03:24:40 dist-test-slave-00qs krb5kdc[23748](info): commencing operation
krb5kdc: starting...
W20260924 03:24:41.533306 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.002s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:24:41 dist-test-slave-00qs krb5kdc[23748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:24:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:24:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:24:41.824476 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:46013
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:46013,127.10.12.189:46305,127.10.12.188:35409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:24:42.267383 23761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:24:42.267867 23761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:24:42.268074 23761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:24:42.277196 23761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:24:42.277401 23761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:24:42.277494 23761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:24:42.277585 23761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:24:42.277648 23761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:24:42.290735 23761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:46013,127.10.12.189:46305,127.10.12.188:35409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:46013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:24:42.293162 23761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:24:42.295140 23761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:24:42.310732 23767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:42.311365 23766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:42.311380 23769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:42.312254 23761 server_base.cc:1034] running on GCE node
I20260924 03:24:42.313453 23761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:24:42.318265 23761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:24:42.322351 23761 hybrid_clock.cc:648] HybridClock initialized: now 1790220282321975 us; error 333 us; skew 500 ppm
I20260924 03:24:42.323606 23761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:24:42 dist-test-slave-00qs krb5kdc[23748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220282, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:24:42.329711 23761 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:24:42.332316 23761 webserver.cc:460] Webserver started at http://127.10.12.190:35389/ using document root <none> and password file <none>
I20260924 03:24:42.333284 23761 fs_manager.cc:359] Metadata directory not provided
I20260924 03:24:42.333442 23761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:24:42.333832 23761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:24:42.338109 23761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "e08f0a864d794234a90d35fe95d84609"
format_stamp: "Formatted at 2026-09-24 03:24:42 on dist-test-slave-00qs"
server_key: "e6f5d75d478f9601f953a50f5971dae0"
server_key_iv: "397186962fa94d59c9f0321847dde34f"
server_key_version: "encryptionkey@0"
I20260924 03:24:42.339303 23761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "e08f0a864d794234a90d35fe95d84609"
format_stamp: "Formatted at 2026-09-24 03:24:42 on dist-test-slave-00qs"
server_key: "e6f5d75d478f9601f953a50f5971dae0"
server_key_iv: "397186962fa94d59c9f0321847dde34f"
server_key_version: "encryptionkey@0"
I20260924 03:24:42.347007 23761 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260924 03:24:42.351665 23776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:42.353422 23761 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:24:42.353627 23761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "e08f0a864d794234a90d35fe95d84609"
format_stamp: "Formatted at 2026-09-24 03:24:42 on dist-test-slave-00qs"
server_key: "e6f5d75d478f9601f953a50f5971dae0"
server_key_iv: "397186962fa94d59c9f0321847dde34f"
server_key_version: "encryptionkey@0"
I20260924 03:24:42.353943 23761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:24:42.388019 23761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:24:42.394109 23761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:24:42.394479 23761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:24:42.418387 23761 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:46013
I20260924 03:24:42.418401 23827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:46013 every 8 connection(s)
I20260924 03:24:42.420496 23761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:24:42.425915 23828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:24:42.429410 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 23761
I20260924 03:24:42.429702 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:24:42.430744 10290 external_mini_cluster.cc:1428] Setting key ccdffd776da5bc2bd3798f25735bf0ca
I20260924 03:24:42.437938 23828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46013 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.12.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.189@KRBTEST.COM" created.
Sep 24 03:24:42 dist-test-slave-00qs krb5kdc[23748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220282, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:24:42.475736 23831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:42.443437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39272 (local address 127.10.12.190:46013)
0924 03:24:42.444840 (+ 1403us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:42.444865 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:42.445398 (+ 533us) server_negotiation.cc:407] Connection header received
0924 03:24:42.448078 (+ 2680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:42.448086 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:42.448439 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:42.448694 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:42.450362 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:42.451844 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:42.453599 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:42.454381 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:42.454605 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:42.465476 (+ 10871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:42.465492 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:42.465518 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:42.465582 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:42.469875 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:42.471016 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:42.471027 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:42.471030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:42.471310 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:42.471871 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:42.471876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:42.471878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:42.472711 (+ 833us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:42.472922 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:42.473702 (+ 780us) server_negotiation.cc:299] Negotiation successful
0924 03:24:42.474227 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":1010,"thread_start_us":707,"threads_started":1}
I20260924 03:24:42.475759 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:42.443437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:46013 (local address 127.0.0.1:39272)
0924 03:24:42.445191 (+ 1754us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:42.445241 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:42.447493 (+ 2252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:42.448720 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:42.448728 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:42.449652 (+ 924us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:24:42.450102 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:42.450127 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:42.452076 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:42.452081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:42.453423 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:42.453440 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:42.454667 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:42.454675 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:42.454863 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:42.461697 (+ 6834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:24:42.461749 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:24:42.465057 (+ 3308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:24:42.470143 (+ 5086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:42.470159 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:42.470173 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:42.470812 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:42.471464 (+ 652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:42.471469 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:42.471471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:42.471724 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:42.472892 (+ 1168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:24:42.472918 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:24:42.473214 (+ 296us) client_negotiation.cc:769] Sending connection context
0924 03:24:42.473536 (+ 322us) client_negotiation.cc:240] Negotiation successful
0924 03:24:42.474384 (+ 848us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1467,"mutex_wait_us":203,"spinlock_wait_cycles":19712,"thread_start_us":1172,"threads_started":1}
I20260924 03:24:42.480242 23828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:24:42.481948 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:42.481354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:46305 (local address 127.0.0.1:38970)
0924 03:24:42.481596 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:42.481748 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:46305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260924 03:24:42.482419 23780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.12.189:46305: connect: Connection refused (error 111)
I20260924 03:24:42.485383 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:42.484584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:46305 (local address 127.0.0.1:38976)
0924 03:24:42.484930 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:42.485125 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:46305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 03:24:42.486476 23828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.189:46305: Network error: Client connection negotiation failed: client connection to 127.10.12.189:46305: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.189.keytab.
Entry for principal kudu/127.10.12.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.189.keytab.
I20260924 03:24:42.501829 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.189:46305
--webserver_interface=127.10.12.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.189.keytab
--principal=kudu/127.10.12.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:46013,127.10.12.189:46305,127.10.12.188:35409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:24:42.536029 23828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } attempt: 1
I20260924 03:24:42.537668 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:42.536958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:46305 (local address 127.0.0.1:38982)
0924 03:24:42.537243 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:42.537439 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:46305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 03:24:42.539018 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:42.538526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:46305 (local address 127.0.0.1:38998)
0924 03:24:42.538695 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:42.538801 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:46305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 03:24:42.539299 23828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.189:46305: Network error: Client connection negotiation failed: client connection to 127.10.12.189:46305: connect: Connection refused (error 111)
I20260924 03:24:42.607681 23828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } attempt: 2
I20260924 03:24:42.609607 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:42.608822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:46305 (local address 127.0.0.1:39004)
0924 03:24:42.609172 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:42.609383 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:46305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260924 03:24:42.611248 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:42.610607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:46305 (local address 127.0.0.1:39016)
0924 03:24:42.610869 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:42.610985 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:46305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 03:24:42.611694 23828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.189:46305: Network error: Client connection negotiation failed: client connection to 127.10.12.189:46305: connect: Connection refused (error 111)
I20260924 03:24:42.703080 23828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } attempt: 3
I20260924 03:24:42.704824 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:42.704207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:46305 (local address 127.0.0.1:39018)
0924 03:24:42.704471 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:42.704615 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:46305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 03:24:42.706355 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:42.705808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:46305 (local address 127.0.0.1:39032)
0924 03:24:42.706000 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:42.706127 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:46305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 03:24:42.706619 23828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.189:46305: Network error: Client connection negotiation failed: client connection to 127.10.12.189:46305: connect: Connection refused (error 111)
I20260924 03:24:42.849917 23828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } attempt: 4
I20260924 03:24:42.851691 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:42.850919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:46305 (local address 127.0.0.1:39038)
0924 03:24:42.851299 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:42.851462 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:46305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260924 03:24:42.853331 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:42.852668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:46305 (local address 127.0.0.1:39050)
0924 03:24:42.852867 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:42.852992 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:46305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 03:24:42.853637 23828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.189:46305: Network error: Client connection negotiation failed: client connection to 127.10.12.189:46305: connect: Connection refused (error 111)
W20260924 03:24:42.962975 23835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:24:42.963392 23835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:24:42.963526 23835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:24:42.971995 23835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:24:42.972193 23835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:24:42.972283 23835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:24:42.972373 23835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:24:42.972436 23835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:24:42.984618 23835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:46013,127.10.12.189:46305,127.10.12.188:35409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.189.keytab
--principal=kudu/127.10.12.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.189:46305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.12.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:24:42.986523 23835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:24:42.988369 23835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:24:43.002902 23841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:43.003262 23840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:43.003556 23843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:43.003984 23835 server_base.cc:1034] running on GCE node
I20260924 03:24:43.005005 23835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:24:43.006881 23835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:24:43.008165 23835 hybrid_clock.cc:648] HybridClock initialized: now 1790220283008060 us; error 45 us; skew 500 ppm
I20260924 03:24:43.008738 23835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:24:43 dist-test-slave-00qs krb5kdc[23748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220283, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:24:43.013243 23835 init.cc:377] Logged in from keytab as kudu/127.10.12.189@KRBTEST.COM (short username kudu)
I20260924 03:24:43.015252 23835 webserver.cc:460] Webserver started at http://127.10.12.189:41381/ using document root <none> and password file <none>
I20260924 03:24:43.016394 23835 fs_manager.cc:359] Metadata directory not provided
I20260924 03:24:43.016558 23835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:24:43.017004 23835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:24:43.021134 23835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-1/data/instance:
uuid: "3192e812fa5b419d95d910e8c80f51fc"
format_stamp: "Formatted at 2026-09-24 03:24:43 on dist-test-slave-00qs"
server_key: "5d18c4ea84ec32c6a23eedcf2869309c"
server_key_iv: "1873d9053c4d792cc663618d2ba9e1e7"
server_key_version: "encryptionkey@0"
I20260924 03:24:43.022169 23835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-1/wal/instance:
uuid: "3192e812fa5b419d95d910e8c80f51fc"
format_stamp: "Formatted at 2026-09-24 03:24:43 on dist-test-slave-00qs"
server_key: "5d18c4ea84ec32c6a23eedcf2869309c"
server_key_iv: "1873d9053c4d792cc663618d2ba9e1e7"
server_key_version: "encryptionkey@0"
I20260924 03:24:43.030503 23835 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20260924 03:24:43.035632 23850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:43.037637 23835 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 03:24:43.037882 23835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-1/wal
uuid: "3192e812fa5b419d95d910e8c80f51fc"
format_stamp: "Formatted at 2026-09-24 03:24:43 on dist-test-slave-00qs"
server_key: "5d18c4ea84ec32c6a23eedcf2869309c"
server_key_iv: "1873d9053c4d792cc663618d2ba9e1e7"
server_key_version: "encryptionkey@0"
I20260924 03:24:43.038133 23835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:24:43.056471 23835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:24:43.062698 23835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:24:43.063047 23835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:24:43.085100 23835 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.189:46305
I20260924 03:24:43.085115 23901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.189:46305 every 8 connection(s)
I20260924 03:24:43.086927 23835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-1/data/info.pb
I20260924 03:24:43.092242 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:24:43.095177 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 23835
I20260924 03:24:43.095410 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-1/wal/instance
I20260924 03:24:43.095920 10290 external_mini_cluster.cc:1428] Setting key 7732eec0aec618ec8814c7e502431ab6
I20260924 03:24:43.102873 23902 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46013 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.12.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.188@KRBTEST.COM" created.
Sep 24 03:24:43 dist-test-slave-00qs krb5kdc[23748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220283, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.189@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:24:43.136714 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.107583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39278 (local address 127.10.12.190:46013)
0924 03:24:43.108448 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:43.108456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:43.110665 (+ 2209us) server_negotiation.cc:407] Connection header received
0924 03:24:43.114047 (+ 3382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:43.114056 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:43.114191 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:43.114379 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:43.117733 (+ 3354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.119110 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.121610 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.122196 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.122339 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:43.127745 (+ 5406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:43.127755 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:43.127760 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:43.127836 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:43.132039 (+ 4203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.133245 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.133251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.133255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.133445 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.134041 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.134047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.134050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.134517 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:43.134687 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:43.135857 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0924 03:24:43.136281 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":386,"threads_started":1}
I20260924 03:24:43.138120 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.108742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:46013 (local address 127.0.0.1:39278)
0924 03:24:43.110332 (+ 1590us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.110431 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:43.113457 (+ 3026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:43.114851 (+ 1394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:43.114873 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:43.116327 (+ 1454us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:24:43.117368 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:43.117400 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.119470 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.119475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:43.121336 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:43.121368 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.122540 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.122547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:43.122848 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:43.123824 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:24:43.123857 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:24:43.127269 (+ 3412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:24:43.132328 (+ 5059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:43.132341 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:43.132364 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:43.133053 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:43.133590 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:43.133595 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:43.133597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:43.133915 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:43.134698 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:24:43.134775 (+ 77us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:24:43.135323 (+ 548us) client_negotiation.cc:769] Sending connection context
0924 03:24:43.135660 (+ 337us) client_negotiation.cc:240] Negotiation successful
0924 03:24:43.136445 (+ 785us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"thread_start_us":719,"threads_started":1}
I20260924 03:24:43.140731 23902 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:24:43 dist-test-slave-00qs krb5kdc[23748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220283, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.189@KRBTEST.COM for kudu/127.10.12.189@KRBTEST.COM
I20260924 03:24:43.153010 23828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } attempt: 5
I20260924 03:24:43.161692 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.141772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:46305 (local address 127.0.0.1:39052)
0924 03:24:43.142034 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.142056 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:43.142207 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:43.143465 (+ 1258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:43.143471 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:43.144110 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:24:43.144312 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:43.144326 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.145876 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.145881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:43.147177 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:43.147193 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.148336 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.148349 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:43.148557 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:43.149314 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:24:43.149340 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:24:43.152285 (+ 2945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:24:43.157258 (+ 4973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:43.157264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:43.157269 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:43.157980 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:43.158465 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:43.158469 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:43.158471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:43.158984 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:43.160039 (+ 1055us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:24:43.160045 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:24:43.160345 (+ 300us) client_negotiation.cc:769] Sending connection context
0924 03:24:43.160517 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 03:24:43.161231 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 03:24:43.161828 23908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.142037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39052 (local address 127.10.12.189:46305)
0924 03:24:43.142919 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:43.142926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:43.142956 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:24:43.143015 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:43.143023 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:43.143294 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:43.143440 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:43.144496 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.145717 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.147348 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.148143 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.148258 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:43.152602 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:43.152631 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:43.152656 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:43.152699 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:43.157118 (+ 4419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.158133 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.158144 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.158148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.158346 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.159175 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.159181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.159185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.159823 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:43.160161 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:43.160719 (+ 558us) server_negotiation.cc:299] Negotiation successful
0924 03:24:43.161438 (+ 719us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":226,"server-negotiator.queue_time_us":693,"thread_start_us":506,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.188.keytab.
Entry for principal kudu/127.10.12.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.188.keytab.
Sep 24 03:24:43 dist-test-slave-00qs krb5kdc[23748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220282, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for kudu/127.10.12.189@KRBTEST.COM
I20260924 03:24:43.166006 23902 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:24:43.166229 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.188:35409
--webserver_interface=127.10.12.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.188.keytab
--principal=kudu/127.10.12.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:46013,127.10.12.189:46305,127.10.12.188:35409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:24:43.167588 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.166927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35409 (local address 127.0.0.1:41186)
0924 03:24:43.167167 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.167341 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260924 03:24:43.168011 23854 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
I20260924 03:24:43.170902 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.170404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35409 (local address 127.0.0.1:41196)
0924 03:24:43.170601 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.170720 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 03:24:43.171451 23902 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:35409: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
I20260924 03:24:43.174804 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.153877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:46305 (local address 127.0.0.1:39060)
0924 03:24:43.154121 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.154144 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:43.154293 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:43.155797 (+ 1504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:43.155804 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:43.156393 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:24:43.156687 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:43.156705 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.158487 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.158493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:43.160298 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:43.160325 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.161428 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.161437 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:43.161631 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:43.162631 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:24:43.162655 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:24:43.166004 (+ 3349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:24:43.171369 (+ 5365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:43.171377 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:43.171381 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:43.172150 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:43.172743 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:43.172747 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:43.172750 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:43.173016 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:43.173744 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:24:43.173748 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:24:43.174011 (+ 263us) client_negotiation.cc:769] Sending connection context
0924 03:24:43.174153 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 03:24:43.174515 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 03:24:43.175020 23909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.154093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39060 (local address 127.10.12.189:46305)
0924 03:24:43.155340 (+ 1247us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:43.155345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:43.155370 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:24:43.155434 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:43.155439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:43.155524 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:43.155693 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:43.156935 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.158313 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.160470 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.161214 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.161350 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:43.166455 (+ 5105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:43.166469 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:43.166475 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:43.166532 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:43.171115 (+ 4583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.172338 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.172345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.172348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.172548 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.173153 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.173157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.173160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.173595 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:43.173764 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:43.174316 (+ 552us) server_negotiation.cc:299] Negotiation successful
0924 03:24:43.174611 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1090,"thread_start_us":799,"threads_started":1}
I20260924 03:24:43.176400 23828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:24:43.178056 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.177497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35409 (local address 127.0.0.1:41212)
0924 03:24:43.177731 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.177844 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 03:24:43.179317 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.178868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35409 (local address 127.0.0.1:41222)
0924 03:24:43.179066 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.179159 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260924 03:24:43.179565 23828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:35409: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
I20260924 03:24:43.220950 23902 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } attempt: 1
I20260924 03:24:43.222792 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.222055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35409 (local address 127.0.0.1:41238)
0924 03:24:43.222374 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.222525 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260924 03:24:43.224231 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.223706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35409 (local address 127.0.0.1:41254)
0924 03:24:43.223942 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.224080 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260924 03:24:43.224509 23902 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:35409: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
I20260924 03:24:43.230906 23828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } attempt: 1
I20260924 03:24:43.232695 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.232049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35409 (local address 127.0.0.1:41258)
0924 03:24:43.232309 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.232461 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 03:24:43.233946 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.233499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35409 (local address 127.0.0.1:41268)
0924 03:24:43.233677 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.233777 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 03:24:43.234222 23828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:35409: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
I20260924 03:24:43.292766 23902 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } attempt: 2
I20260924 03:24:43.294371 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.293736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35409 (local address 127.0.0.1:41282)
0924 03:24:43.294028 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.294149 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260924 03:24:43.295698 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.295265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35409 (local address 127.0.0.1:41296)
0924 03:24:43.295446 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.295543 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 03:24:43.295939 23902 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:35409: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
I20260924 03:24:43.302511 23828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } attempt: 2
I20260924 03:24:43.304028 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.303335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35409 (local address 127.0.0.1:41308)
0924 03:24:43.303602 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.303773 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 03:24:43.305339 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.304860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35409 (local address 127.0.0.1:41318)
0924 03:24:43.305067 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.305178 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260924 03:24:43.305673 23828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:35409: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
I20260924 03:24:43.387270 23902 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } attempt: 3
I20260924 03:24:43.388902 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.388237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35409 (local address 127.0.0.1:41330)
0924 03:24:43.388507 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.388624 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 03:24:43.390285 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.389740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35409 (local address 127.0.0.1:41346)
0924 03:24:43.390028 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.390122 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
W20260924 03:24:43.390535 23902 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:35409: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
I20260924 03:24:43.412005 23828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } attempt: 3
I20260924 03:24:43.413427 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.412852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35409 (local address 127.0.0.1:41350)
0924 03:24:43.413090 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.413203 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 03:24:43.414783 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.414369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35409 (local address 127.0.0.1:41358)
0924 03:24:43.414550 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.414646 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 03:24:43.415016 23828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:35409: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
I20260924 03:24:43.533849 23902 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } attempt: 4
I20260924 03:24:43.535488 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.534902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35409 (local address 127.0.0.1:41372)
0924 03:24:43.535151 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.535262 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 03:24:43.536891 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.536442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35409 (local address 127.0.0.1:41378)
0924 03:24:43.536630 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.536726 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 03:24:43.537143 23902 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:35409: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
I20260924 03:24:43.592278 23828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } attempt: 4
I20260924 03:24:43.593940 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.593365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35409 (local address 127.0.0.1:41386)
0924 03:24:43.593621 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.593734 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 03:24:43.595135 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.594702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35409 (local address 127.0.0.1:41394)
0924 03:24:43.594871 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.594979 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 03:24:43.595350 23828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:35409: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35409: connect: Connection refused (error 111)
W20260924 03:24:43.599176 23911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:24:43.599637 23911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:24:43.599797 23911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:24:43.608814 23911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:24:43.608994 23911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:24:43.609079 23911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:24:43.609172 23911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:24:43.609239 23911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:24:43.623809 23911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:46013,127.10.12.189:46305,127.10.12.188:35409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.188.keytab
--principal=kudu/127.10.12.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.188:35409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.12.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:24:43.625774 23911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:24:43.627785 23911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:24:43.645571 23917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:43.645606 23919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:43.645709 23916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:43.646612 23911 server_base.cc:1034] running on GCE node
I20260924 03:24:43.647794 23911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:24:43.650265 23911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:24:43.651643 23911 hybrid_clock.cc:648] HybridClock initialized: now 1790220283651524 us; error 109 us; skew 500 ppm
I20260924 03:24:43.652277 23911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:24:43 dist-test-slave-00qs krb5kdc[23748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220283, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:24:43.656553 23911 init.cc:377] Logged in from keytab as kudu/127.10.12.188@KRBTEST.COM (short username kudu)
I20260924 03:24:43.659157 23911 webserver.cc:460] Webserver started at http://127.10.12.188:43835/ using document root <none> and password file <none>
I20260924 03:24:43.660180 23911 fs_manager.cc:359] Metadata directory not provided
I20260924 03:24:43.660336 23911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:24:43.660764 23911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:24:43.665071 23911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-2/data/instance:
uuid: "e1c240f4458a45868ad9a5237557b656"
format_stamp: "Formatted at 2026-09-24 03:24:43 on dist-test-slave-00qs"
server_key: "23e7f050c8a3a4a3f3aca28c09727d67"
server_key_iv: "227e3d0139f6b0a0089300520c7f14b2"
server_key_version: "encryptionkey@0"
I20260924 03:24:43.666216 23911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-2/wal/instance:
uuid: "e1c240f4458a45868ad9a5237557b656"
format_stamp: "Formatted at 2026-09-24 03:24:43 on dist-test-slave-00qs"
server_key: "23e7f050c8a3a4a3f3aca28c09727d67"
server_key_iv: "227e3d0139f6b0a0089300520c7f14b2"
server_key_version: "encryptionkey@0"
I20260924 03:24:43.674166 23911 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260924 03:24:43.679376 23926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:43.681833 23911 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260924 03:24:43.682096 23911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-2/wal
uuid: "e1c240f4458a45868ad9a5237557b656"
format_stamp: "Formatted at 2026-09-24 03:24:43 on dist-test-slave-00qs"
server_key: "23e7f050c8a3a4a3f3aca28c09727d67"
server_key_iv: "227e3d0139f6b0a0089300520c7f14b2"
server_key_version: "encryptionkey@0"
I20260924 03:24:43.682371 23911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:24:43.714370 23911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:24:43.720072 23911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:24:43.720420 23911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:24:43.742038 23911 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.188:35409
I20260924 03:24:43.742060 23977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.188:35409 every 8 connection(s)
I20260924 03:24:43.743989 23911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-2/data/info.pb
I20260924 03:24:43.749266 23978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:24:43.753259 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 23911
I20260924 03:24:43.753490 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790220067268388-10290-0/minicluster-data/master-2/wal/instance
I20260924 03:24:43.754047 10290 external_mini_cluster.cc:1428] Setting key 09cdda7ae2898e89d98688a62358574d
I20260924 03:24:43.758862 23978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46013 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:24:43 dist-test-slave-00qs krb5kdc[23748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.188@KRBTEST.COM
Sep 24 03:24:43 dist-test-slave-00qs krb5kdc[23748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220283, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.188@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:24:43.789000 23983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.766170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39286 (local address 127.10.12.190:46013)
0924 03:24:43.768309 (+ 2139us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:43.768316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:43.768340 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:24:43.768419 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:43.768425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:43.768516 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:43.768651 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:43.770848 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.772040 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.774225 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.774732 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.774854 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:43.780207 (+ 5353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:43.780215 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:43.780219 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:43.780288 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:43.783921 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.785485 (+ 1564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.785492 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.785496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.785741 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.786331 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.786336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.786339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.786814 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:43.786979 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:43.788007 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0924 03:24:43.788426 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1827,"thread_start_us":1630,"threads_started":1}
I20260924 03:24:43.790562 23982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.766055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:46013 (local address 127.0.0.1:39286)
0924 03:24:43.767401 (+ 1346us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.767476 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:43.767758 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:43.768907 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:43.768916 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:43.770080 (+ 1164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:24:43.770563 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:43.770583 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.772265 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.772270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:43.773974 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:43.773995 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.775054 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.775063 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:43.775309 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:43.776341 (+ 1032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:24:43.776382 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:24:43.779714 (+ 3332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:24:43.784425 (+ 4711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:43.784442 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:43.784460 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:43.785294 (+ 834us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:43.785884 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:43.785888 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:43.785892 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:43.786144 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:43.786952 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:24:43.786959 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:24:43.787380 (+ 421us) client_negotiation.cc:769] Sending connection context
0924 03:24:43.787805 (+ 425us) client_negotiation.cc:240] Negotiation successful
0924 03:24:43.788634 (+ 829us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"mutex_wait_us":4,"spinlock_wait_cycles":24832,"thread_start_us":685,"threads_started":1}
I20260924 03:24:43.790592 23981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.757182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41402 (local address 127.10.12.188:35409)
0924 03:24:43.758613 (+ 1431us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:43.758630 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:43.758702 (+ 72us) server_negotiation.cc:407] Connection header received
0924 03:24:43.762179 (+ 3477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:43.762229 (+ 50us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:43.763123 (+ 894us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:43.763677 (+ 554us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:43.766266 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.768972 (+ 2706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.771244 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.772545 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.772715 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:43.778657 (+ 5942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:43.778669 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:43.778688 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:43.778739 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:43.783928 (+ 5189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.785484 (+ 1556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.785494 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.785498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.785837 (+ 339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.786309 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.786314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.786317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.787341 (+ 1024us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:43.787621 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:43.788443 (+ 822us) server_negotiation.cc:299] Negotiation successful
0924 03:24:43.788775 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":1076,"thread_start_us":752,"threads_started":1}
I20260924 03:24:43.792752 23978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:24:43 dist-test-slave-00qs krb5kdc[23748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220283, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.188@KRBTEST.COM for kudu/127.10.12.189@KRBTEST.COM
Sep 24 03:24:43 dist-test-slave-00qs krb5kdc[23748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:24:43.811225 23982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.793740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:46305 (local address 127.0.0.1:39064)
0924 03:24:43.793948 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.793970 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:43.794113 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:43.794756 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:43.794761 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:43.795257 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:24:43.795507 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:43.795520 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.796683 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.796688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:43.797879 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:43.797897 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.798836 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.798844 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:43.799103 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:43.800071 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:24:43.800095 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:24:43.802809 (+ 2714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:24:43.807731 (+ 4922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:43.807737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:43.807742 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:43.808450 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:43.809070 (+ 620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:43.809075 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:43.809078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:43.809280 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:43.810065 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:24:43.810071 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:24:43.810285 (+ 214us) client_negotiation.cc:769] Sending connection context
0924 03:24:43.810539 (+ 254us) client_negotiation.cc:240] Negotiation successful
0924 03:24:43.810899 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 03:24:43.811444 23984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.793729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39064 (local address 127.10.12.189:46305)
0924 03:24:43.794350 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:43.794356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:43.794377 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:24:43.794453 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:43.794459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:43.794551 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:43.794691 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:43.795699 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.796544 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.798047 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.798623 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.798765 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:43.803136 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:43.803144 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:43.803149 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:43.803209 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:43.807449 (+ 4240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.808651 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.808659 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.808663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.808884 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.809419 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.809425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.809428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.809908 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:43.810050 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:43.810716 (+ 666us) server_negotiation.cc:299] Negotiation successful
0924 03:24:43.811022 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":351,"threads_started":1}
I20260924 03:24:43.812645 23978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:24:43.813162 23983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.794626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39302 (local address 127.10.12.190:46013)
0924 03:24:43.794829 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:43.794834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:43.794851 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:24:43.794920 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:43.794923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:43.794996 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:43.795073 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:43.796631 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.797475 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.799214 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.799867 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.800031 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:43.805675 (+ 5644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:43.805687 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:43.805692 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:43.805762 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:43.809588 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.810713 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.810720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.810723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.811000 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.811486 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.811492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.811495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.811893 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:43.812254 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:43.812584 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 03:24:43.812808 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
Sep 24 03:24:43 dist-test-slave-00qs krb5kdc[23748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220283, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.188@KRBTEST.COM for kudu/127.10.12.188@KRBTEST.COM
Sep 24 03:24:43 dist-test-slave-00qs krb5kdc[23748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.189@KRBTEST.COM
I20260924 03:24:43.828261 23982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.813504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35409 (local address 127.0.0.1:41408)
0924 03:24:43.813692 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.813712 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:43.813834 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:43.814199 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:43.814202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:43.814727 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:24:43.815006 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:43.815019 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.816153 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.816158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:43.817268 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:43.817282 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.818030 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.818035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:43.818235 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:43.818885 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:24:43.818904 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:24:43.821753 (+ 2849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:24:43.825488 (+ 3735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:43.825494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:43.825499 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:43.825958 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:43.826335 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:43.826341 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:43.826345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:43.826486 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:43.827155 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:24:43.827159 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:24:43.827356 (+ 197us) client_negotiation.cc:769] Sending connection context
0924 03:24:43.827490 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 03:24:43.827852 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 03:24:43.828467 23981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.813697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41408 (local address 127.10.12.188:35409)
0924 03:24:43.813867 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:43.813874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:43.813891 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:24:43.813946 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:43.813951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:43.814029 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:43.814131 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:43.815141 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.816051 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.817414 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.817886 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.817999 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:43.821997 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:43.822003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:43.822007 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:43.822046 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:43.825313 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.826072 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.826078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.826082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.826227 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.826587 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.826591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.826593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.826992 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:43.827165 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:43.827614 (+ 449us) server_negotiation.cc:299] Negotiation successful
0924 03:24:43.828082 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":129,"server-negotiator.queue_time_us":48}
I20260924 03:24:43.834154 23984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.815449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39074 (local address 127.10.12.189:46305)
0924 03:24:43.815732 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:43.815737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:43.816025 (+ 288us) server_negotiation.cc:407] Connection header received
0924 03:24:43.816289 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:43.816303 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:43.816397 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:43.816501 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:43.818244 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.819016 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.820763 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.821330 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.821464 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:43.826923 (+ 5459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:43.826933 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:43.826938 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:43.826994 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:43.830843 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.831788 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.831796 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.831800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.832044 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.832487 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.832492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.832495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.832907 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:43.833215 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:43.833506 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 03:24:43.833786 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 03:24:43.835400 23978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1c240f4458a45868ad9a5237557b656: Bootstrap starting.
I20260924 03:24:43.836405 23902 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } attempt: 5
I20260924 03:24:43.840317 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:24:43.840956 23978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1c240f4458a45868ad9a5237557b656: Neither blocks nor log segments found. Creating new log.
I20260924 03:24:43.842867 23978 log.cc:824] T 00000000000000000000000000000000 P e1c240f4458a45868ad9a5237557b656: Log is configured to *not* fsync() on all Append() calls
Sep 24 03:24:43 dist-test-slave-00qs krb5kdc[23748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220283, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.189@KRBTEST.COM for kudu/127.10.12.188@KRBTEST.COM
I20260924 03:24:43.853643 23978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1c240f4458a45868ad9a5237557b656: No bootstrap required, opened a new log
I20260924 03:24:43.855927 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.837094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35409 (local address 127.0.0.1:41410)
0924 03:24:43.837372 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.837392 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:43.837501 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:43.837871 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:43.837877 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:43.838419 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:24:43.838655 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:43.838669 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.840085 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.840092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:43.841379 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:43.841396 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.842848 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.842855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:43.843099 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:43.844102 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:24:43.844134 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:24:43.847672 (+ 3538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:24:43.852438 (+ 4766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:43.852444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:43.852448 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:43.853084 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:43.853611 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:43.853617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:43.853620 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:43.853828 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:43.854713 (+ 885us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:24:43.854718 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:24:43.854923 (+ 205us) client_negotiation.cc:769] Sending connection context
0924 03:24:43.855098 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 03:24:43.855483 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260924 03:24:43.857402 23981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.837150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41410 (local address 127.10.12.188:35409)
0924 03:24:43.837495 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:43.837502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:43.837522 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:24:43.837641 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:43.837647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:43.837731 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:43.837828 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:43.838849 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.839856 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.841634 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.842545 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.842815 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:43.848046 (+ 5231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:43.848057 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:43.848063 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:43.848128 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:43.852193 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.853261 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.853267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.853271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.853466 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.853961 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.853965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.853968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.854454 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:43.854621 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:43.856571 (+ 1950us) server_negotiation.cc:299] Negotiation successful
0924 03:24:43.856898 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260924 03:24:43.865890 23902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3192e812fa5b419d95d910e8c80f51fc: Bootstrap starting.
I20260924 03:24:43.869158 23978 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1c240f4458a45868ad9a5237557b656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f0a864d794234a90d35fe95d84609" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46013 } } peers { permanent_uuid: "3192e812fa5b419d95d910e8c80f51fc" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } } peers { permanent_uuid: "e1c240f4458a45868ad9a5237557b656" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } }
I20260924 03:24:43.869974 23978 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1c240f4458a45868ad9a5237557b656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:24:43.870139 23978 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1c240f4458a45868ad9a5237557b656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1c240f4458a45868ad9a5237557b656, State: Initialized, Role: FOLLOWER
I20260924 03:24:43.871236 23978 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1c240f4458a45868ad9a5237557b656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f0a864d794234a90d35fe95d84609" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46013 } } peers { permanent_uuid: "3192e812fa5b419d95d910e8c80f51fc" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } } peers { permanent_uuid: "e1c240f4458a45868ad9a5237557b656" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } }
I20260924 03:24:43.872608 23828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } attempt: 5
I20260924 03:24:43.873160 23902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3192e812fa5b419d95d910e8c80f51fc: Neither blocks nor log segments found. Creating new log.
I20260924 03:24:43.875224 23902 log.cc:824] T 00000000000000000000000000000000 P 3192e812fa5b419d95d910e8c80f51fc: Log is configured to *not* fsync() on all Append() calls
I20260924 03:24:43.881011 23993 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1c240f4458a45868ad9a5237557b656 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f0a864d794234a90d35fe95d84609" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46013 } } peers { permanent_uuid: "3192e812fa5b419d95d910e8c80f51fc" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } } peers { permanent_uuid: "e1c240f4458a45868ad9a5237557b656" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } } }
I20260924 03:24:43.881644 23993 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1c240f4458a45868ad9a5237557b656 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:24:43.884438 23978 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1c240f4458a45868ad9a5237557b656 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:24:43.899042 23984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.848689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.10.12.189:46305)
0924 03:24:43.848911 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:43.848917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:43.866951 (+ 18034us) server_negotiation.cc:407] Connection header received
0924 03:24:43.868562 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:43.868572 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:43.868717 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:43.868835 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:43.881221 (+ 12386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.882487 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.885334 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.886082 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.886217 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:43.890224 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:43.890234 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:43.890239 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:43.890299 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:43.894529 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.895885 (+ 1356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.895892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.895896 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.896177 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.896676 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.896682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.896686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.897141 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:43.897468 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:43.898064 (+ 596us) server_negotiation.cc:299] Negotiation successful
0924 03:24:43.898575 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 03:24:43.899772 23902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3192e812fa5b419d95d910e8c80f51fc: No bootstrap required, opened a new log
Sep 24 03:24:43 dist-test-slave-00qs krb5kdc[23748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220282, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for kudu/127.10.12.188@KRBTEST.COM
I20260924 03:24:43.905300 23983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.849170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39312 (local address 127.10.12.190:46013)
0924 03:24:43.849383 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:43.849391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:43.862626 (+ 13235us) server_negotiation.cc:407] Connection header received
0924 03:24:43.863449 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:43.863459 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:43.863587 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:43.863677 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:43.870380 (+ 6703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.871812 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.878171 (+ 6359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.879179 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.879365 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:43.885472 (+ 6107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:43.885481 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:43.885486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:43.885547 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:43.890154 (+ 4607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.892445 (+ 2291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.892452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.892455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.892680 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.894201 (+ 1521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.894208 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.894213 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.895120 (+ 907us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:43.895569 (+ 449us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:43.904378 (+ 8809us) server_negotiation.cc:299] Negotiation successful
0924 03:24:43.904797 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 03:24:43.914577 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.873486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35409 (local address 127.0.0.1:41422)
0924 03:24:43.873864 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:43.873888 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:43.874030 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:43.888723 (+ 14693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:43.888731 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:43.889371 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:24:43.889675 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:43.889696 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.895414 (+ 5718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.895422 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:43.896983 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:43.897016 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.900468 (+ 3452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.900477 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:43.900699 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:43.901391 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:24:43.901418 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:24:43.904658 (+ 3240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:24:43.910751 (+ 6093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:43.910759 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:43.910763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:43.911445 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:43.912138 (+ 693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:43.912143 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:43.912147 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:43.912350 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:43.913288 (+ 938us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:24:43.913293 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:24:43.913498 (+ 205us) client_negotiation.cc:769] Sending connection context
0924 03:24:43.913666 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 03:24:43.914068 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220}
I20260924 03:24:43.914741 23981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.867273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41418 (local address 127.10.12.188:35409)
0924 03:24:43.869207 (+ 1934us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:43.869214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:43.882910 (+ 13696us) server_negotiation.cc:407] Connection header received
0924 03:24:43.884478 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:43.884487 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:43.884632 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:43.884730 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:43.891935 (+ 7205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.893097 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.898233 (+ 5136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.899004 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.899137 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:43.906289 (+ 7152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:43.906297 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:43.906302 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:43.906357 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:43.910751 (+ 4394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.911741 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.911745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.911749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.911910 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.912484 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.912489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.912493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.913142 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:43.913424 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:43.914052 (+ 628us) server_negotiation.cc:299] Negotiation successful
0924 03:24:43.914367 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":1780,"spinlock_wait_cycles":7296,"thread_start_us":20975,"threads_started":1}
I20260924 03:24:43.917001 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:43.874422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41422 (local address 127.10.12.188:35409)
0924 03:24:43.887981 (+ 13559us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:43.887987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:43.888015 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:24:43.888083 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:43.888088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:43.888177 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:43.889900 (+ 1723us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:43.890238 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.891706 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.897261 (+ 5555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:43.898048 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:43.903068 (+ 5020us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:43.905104 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:43.905113 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:43.905118 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:43.905170 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:43.910479 (+ 5309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.911669 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.911674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.911678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.912003 (+ 325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:43.912491 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:43.912495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:43.912497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:43.913142 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:43.916050 (+ 2908us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:43.916220 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 03:24:43.916542 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":13334,"spinlock_wait_cycles":95744}
I20260924 03:24:43.919817 23902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3192e812fa5b419d95d910e8c80f51fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f0a864d794234a90d35fe95d84609" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46013 } } peers { permanent_uuid: "3192e812fa5b419d95d910e8c80f51fc" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } } peers { permanent_uuid: "e1c240f4458a45868ad9a5237557b656" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } }
I20260924 03:24:43.921057 23902 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3192e812fa5b419d95d910e8c80f51fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:24:43.921228 23902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3192e812fa5b419d95d910e8c80f51fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3192e812fa5b419d95d910e8c80f51fc, State: Initialized, Role: FOLLOWER
I20260924 03:24:43.922302 23902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3192e812fa5b419d95d910e8c80f51fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f0a864d794234a90d35fe95d84609" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46013 } } peers { permanent_uuid: "3192e812fa5b419d95d910e8c80f51fc" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } } peers { permanent_uuid: "e1c240f4458a45868ad9a5237557b656" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } }
I20260924 03:24:43.925232 23828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609: Bootstrap starting.
I20260924 03:24:43.931397 23902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3192e812fa5b419d95d910e8c80f51fc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:24:43.931010 24002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3192e812fa5b419d95d910e8c80f51fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f0a864d794234a90d35fe95d84609" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46013 } } peers { permanent_uuid: "3192e812fa5b419d95d910e8c80f51fc" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } } peers { permanent_uuid: "e1c240f4458a45868ad9a5237557b656" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } } }
I20260924 03:24:43.931885 24002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3192e812fa5b419d95d910e8c80f51fc [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:24:43.933020 23828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609: Neither blocks nor log segments found. Creating new log.
I20260924 03:24:43.934947 23828 log.cc:824] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609: Log is configured to *not* fsync() on all Append() calls
I20260924 03:24:43.940999 23828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609: No bootstrap required, opened a new log
I20260924 03:24:43.939721 23911 master_runner.cc:186] Error getting master registration for 127.10.12.190:46013: OK, instance_id { permanent_uuid: "e08f0a864d794234a90d35fe95d84609" instance_seqno: 1790220282411429 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:24:43.952983 23911 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:24:43.958869 23828 raft_consensus.cc:348] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f0a864d794234a90d35fe95d84609" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46013 } } peers { permanent_uuid: "3192e812fa5b419d95d910e8c80f51fc" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } } peers { permanent_uuid: "e1c240f4458a45868ad9a5237557b656" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } }
I20260924 03:24:43.959579 23828 raft_consensus.cc:374] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:24:43.959753 23828 raft_consensus.cc:729] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e08f0a864d794234a90d35fe95d84609, State: Initialized, Role: FOLLOWER
I20260924 03:24:43.960706 23828 consensus_queue.cc:260] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f0a864d794234a90d35fe95d84609" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46013 } } peers { permanent_uuid: "3192e812fa5b419d95d910e8c80f51fc" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } } peers { permanent_uuid: "e1c240f4458a45868ad9a5237557b656" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } }
W20260924 03:24:43.961925 24009 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e1c240f4458a45868ad9a5237557b656: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:24:43.962077 24009 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:24:43.962016 24016 sys_catalog.cc:455] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f0a864d794234a90d35fe95d84609" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46013 } } peers { permanent_uuid: "3192e812fa5b419d95d910e8c80f51fc" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } } peers { permanent_uuid: "e1c240f4458a45868ad9a5237557b656" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } } }
I20260924 03:24:43.962873 24016 sys_catalog.cc:458] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:24:43.964006 23828 sys_catalog.cc:564] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 03:24:43.973204 23932 tablet.cc:2369] T 00000000000000000000000000000000 P e1c240f4458a45868ad9a5237557b656: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:24:44.000628 23761 master_runner.cc:186] Error getting master registration for 127.10.12.189:46305: OK, instance_id { permanent_uuid: "3192e812fa5b419d95d910e8c80f51fc" instance_seqno: 1790220283078333 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:24:44.016562 23761 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 03:24:44.019829 24031 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:24:44.020928 24031 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:24:44.022532 23835 master_runner.cc:263] No leader master found from master 3192e812fa5b419d95d910e8c80f51fc
I20260924 03:24:44.022704 23835 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 03:24:44.024297 24032 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3192e812fa5b419d95d910e8c80f51fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:24:44.024456 24032 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 03:24:44.066578 23856 tablet.cc:2369] T 00000000000000000000000000000000 P 3192e812fa5b419d95d910e8c80f51fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:24:44.148931 23782 tablet.cc:2369] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:24:44.315038 24016 raft_consensus.cc:482] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:24:44.315435 24016 raft_consensus.cc:504] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f0a864d794234a90d35fe95d84609" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46013 } } peers { permanent_uuid: "3192e812fa5b419d95d910e8c80f51fc" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } } peers { permanent_uuid: "e1c240f4458a45868ad9a5237557b656" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } }
I20260924 03:24:44.317858 24016 leader_election.cc:288] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3192e812fa5b419d95d910e8c80f51fc (127.10.12.189:46305), e1c240f4458a45868ad9a5237557b656 (127.10.12.188:35409)
I20260924 03:24:44.321051 23877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e08f0a864d794234a90d35fe95d84609" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3192e812fa5b419d95d910e8c80f51fc" is_pre_election: true
I20260924 03:24:44.322472 23877 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3192e812fa5b419d95d910e8c80f51fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e08f0a864d794234a90d35fe95d84609 in term 0.
I20260924 03:24:44.323724 23953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e08f0a864d794234a90d35fe95d84609" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1c240f4458a45868ad9a5237557b656" is_pre_election: true
I20260924 03:24:44.324971 23953 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e1c240f4458a45868ad9a5237557b656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e08f0a864d794234a90d35fe95d84609 in term 0.
I20260924 03:24:44.326303 23780 leader_election.cc:302] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3192e812fa5b419d95d910e8c80f51fc, e08f0a864d794234a90d35fe95d84609; no voters:
I20260924 03:24:44.327371 24016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:24:44.327560 24016 raft_consensus.cc:482] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:24:44.327672 24016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:24:44.331470 24016 raft_consensus.cc:504] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f0a864d794234a90d35fe95d84609" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46013 } } peers { permanent_uuid: "3192e812fa5b419d95d910e8c80f51fc" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } } peers { permanent_uuid: "e1c240f4458a45868ad9a5237557b656" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } }
I20260924 03:24:44.333184 23877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e08f0a864d794234a90d35fe95d84609" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3192e812fa5b419d95d910e8c80f51fc"
I20260924 03:24:44.333624 23877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3192e812fa5b419d95d910e8c80f51fc [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:24:44.337654 23877 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3192e812fa5b419d95d910e8c80f51fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e08f0a864d794234a90d35fe95d84609 in term 1.
I20260924 03:24:44.338938 23780 leader_election.cc:302] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3192e812fa5b419d95d910e8c80f51fc, e08f0a864d794234a90d35fe95d84609; no voters:
I20260924 03:24:44.342619 24016 leader_election.cc:288] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [CANDIDATE]: Term 1 election: Requested vote from peers 3192e812fa5b419d95d910e8c80f51fc (127.10.12.189:46305), e1c240f4458a45868ad9a5237557b656 (127.10.12.188:35409)
I20260924 03:24:44.343472 24016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:24:44.344161 24016 raft_consensus.cc:686] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [term 1 LEADER]: Becoming Leader. State: Replica: e08f0a864d794234a90d35fe95d84609, State: Running, Role: LEADER
I20260924 03:24:44.347772 24016 consensus_queue.cc:237] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f0a864d794234a90d35fe95d84609" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46013 } } peers { permanent_uuid: "3192e812fa5b419d95d910e8c80f51fc" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } } peers { permanent_uuid: "e1c240f4458a45868ad9a5237557b656" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } }
I20260924 03:24:44.358055 23953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e08f0a864d794234a90d35fe95d84609" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1c240f4458a45868ad9a5237557b656"
I20260924 03:24:44.358407 23953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1c240f4458a45868ad9a5237557b656 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:24:44.362653 23953 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e1c240f4458a45868ad9a5237557b656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e08f0a864d794234a90d35fe95d84609 in term 1.
I20260924 03:24:44.369464 24035 sys_catalog.cc:455] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e08f0a864d794234a90d35fe95d84609. Latest consensus state: current_term: 1 leader_uuid: "e08f0a864d794234a90d35fe95d84609" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f0a864d794234a90d35fe95d84609" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46013 } } peers { permanent_uuid: "3192e812fa5b419d95d910e8c80f51fc" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } } peers { permanent_uuid: "e1c240f4458a45868ad9a5237557b656" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } } }
I20260924 03:24:44.369999 24035 sys_catalog.cc:458] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [sys.catalog]: This master's current role is: LEADER
I20260924 03:24:44.374589 24037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:24:44.377878 24037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:24:44.402577 23953 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e1c240f4458a45868ad9a5237557b656 [term 1 FOLLOWER]: Refusing update from remote peer e08f0a864d794234a90d35fe95d84609: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:24:44.403820 24035 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1c240f4458a45868ad9a5237557b656" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:24:44.411598 23877 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3192e812fa5b419d95d910e8c80f51fc [term 1 FOLLOWER]: Refusing update from remote peer e08f0a864d794234a90d35fe95d84609: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:24:44.412709 24035 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3192e812fa5b419d95d910e8c80f51fc" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:24:44.443439 24039 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1c240f4458a45868ad9a5237557b656 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e08f0a864d794234a90d35fe95d84609. Latest consensus state: current_term: 1 leader_uuid: "e08f0a864d794234a90d35fe95d84609" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f0a864d794234a90d35fe95d84609" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46013 } } peers { permanent_uuid: "3192e812fa5b419d95d910e8c80f51fc" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } } peers { permanent_uuid: "e1c240f4458a45868ad9a5237557b656" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } } }
I20260924 03:24:44.443847 24039 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1c240f4458a45868ad9a5237557b656 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:24:44.443709 24002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3192e812fa5b419d95d910e8c80f51fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e08f0a864d794234a90d35fe95d84609. Latest consensus state: current_term: 1 leader_uuid: "e08f0a864d794234a90d35fe95d84609" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f0a864d794234a90d35fe95d84609" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46013 } } peers { permanent_uuid: "3192e812fa5b419d95d910e8c80f51fc" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } } peers { permanent_uuid: "e1c240f4458a45868ad9a5237557b656" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } } }
I20260924 03:24:44.444151 24002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3192e812fa5b419d95d910e8c80f51fc [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:24:44.445866 24038 mvcc.cc:205] Tried to move back new op lower bound from 7332742284894273536 to 7332742284712194048. Current Snapshot: MvccSnapshot[applied={T|T < 7332742284894273536}]
I20260924 03:24:44.447773 24041 mvcc.cc:205] Tried to move back new op lower bound from 7332742284894273536 to 7332742284712194048. Current Snapshot: MvccSnapshot[applied={T|T < 7332742284894273536}]
I20260924 03:24:44.449028 24040 mvcc.cc:205] Tried to move back new op lower bound from 7332742284894273536 to 7332742284712194048. Current Snapshot: MvccSnapshot[applied={T|T < 7332742284894273536}]
I20260924 03:24:44.459144 24002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3192e812fa5b419d95d910e8c80f51fc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e08f0a864d794234a90d35fe95d84609" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f0a864d794234a90d35fe95d84609" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46013 } } peers { permanent_uuid: "3192e812fa5b419d95d910e8c80f51fc" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } } peers { permanent_uuid: "e1c240f4458a45868ad9a5237557b656" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } } }
I20260924 03:24:44.459484 24002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3192e812fa5b419d95d910e8c80f51fc [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:24:44.459930 24016 sys_catalog.cc:455] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e08f0a864d794234a90d35fe95d84609" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f0a864d794234a90d35fe95d84609" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46013 } } peers { permanent_uuid: "3192e812fa5b419d95d910e8c80f51fc" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } } peers { permanent_uuid: "e1c240f4458a45868ad9a5237557b656" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } } }
I20260924 03:24:44.460320 24016 sys_catalog.cc:458] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [sys.catalog]: This master's current role is: LEADER
I20260924 03:24:44.460628 24035 sys_catalog.cc:455] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e08f0a864d794234a90d35fe95d84609" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f0a864d794234a90d35fe95d84609" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46013 } } peers { permanent_uuid: "3192e812fa5b419d95d910e8c80f51fc" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } } peers { permanent_uuid: "e1c240f4458a45868ad9a5237557b656" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } } }
I20260924 03:24:44.460877 24035 sys_catalog.cc:458] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609 [sys.catalog]: This master's current role is: LEADER
I20260924 03:24:44.461498 24037 catalog_manager.cc:1347] Generated new cluster ID: 1540720fa0c34b688ce3530dc169fef5
I20260924 03:24:44.461621 24037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:24:44.464422 24039 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1c240f4458a45868ad9a5237557b656 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e08f0a864d794234a90d35fe95d84609" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08f0a864d794234a90d35fe95d84609" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 46013 } } peers { permanent_uuid: "3192e812fa5b419d95d910e8c80f51fc" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 46305 } } peers { permanent_uuid: "e1c240f4458a45868ad9a5237557b656" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35409 } } }
I20260924 03:24:44.464792 24039 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1c240f4458a45868ad9a5237557b656 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:24:44.487928 24037 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:24:44.490093 24037 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:24:44.515074 24037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e08f0a864d794234a90d35fe95d84609: Generated new TSK 0
I20260924 03:24:44.516460 24037 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:24:44.964182 24009 catalog_manager.cc:1259] Loaded cluster ID: 1540720fa0c34b688ce3530dc169fef5
I20260924 03:24:44.964309 24009 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e1c240f4458a45868ad9a5237557b656: loading cluster ID for follower catalog manager: success
I20260924 03:24:44.967123 24009 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e1c240f4458a45868ad9a5237557b656: acquiring CA information for follower catalog manager: success
I20260924 03:24:44.969074 24009 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e1c240f4458a45868ad9a5237557b656: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:24:45.026533 24032 catalog_manager.cc:1259] Loaded cluster ID: 1540720fa0c34b688ce3530dc169fef5
I20260924 03:24:45.026710 24032 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3192e812fa5b419d95d910e8c80f51fc: loading cluster ID for follower catalog manager: success
I20260924 03:24:45.029485 24032 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3192e812fa5b419d95d910e8c80f51fc: acquiring CA information for follower catalog manager: success
I20260924 03:24:45.031476 24032 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3192e812fa5b419d95d910e8c80f51fc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:24:45.113546 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 23761
I20260924 03:24:45.114886 23823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:24:45.243454 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 23761
I20260924 03:24:45.266122 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 23835
I20260924 03:24:45.267736 23897 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:24:45.393191 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 23835
I20260924 03:24:45.414780 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 23911
I20260924 03:24:45.416234 23973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:24:45.543581 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 23911
2026-09-24T03:24:45Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5089 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:24:45 dist-test-slave-00qs krb5kdc[24055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:24:45 dist-test-slave-00qs krb5kdc[24055](info): set up 2 sockets
Sep 24 03:24:45 dist-test-slave-00qs krb5kdc[24055](info): commencing operation
krb5kdc: starting...
W20260924 03:24:46.629328 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.005s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:24:46 dist-test-slave-00qs krb5kdc[24055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:24:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:24:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:24:46.925832 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:37799
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:37799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:24:47.379276 24068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:24:47.379807 24068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:24:47.379936 24068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:24:47.388811 24068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:24:47.388986 24068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:24:47.389045 24068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:24:47.389101 24068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:24:47.389149 24068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:24:47.400940 24068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:37799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:37799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:24:47.402781 24068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:24:47.404568 24068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:24:47.420499 24073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:47.420558 24074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:47.420522 24076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:47.421732 24068 server_base.cc:1034] running on GCE node
I20260924 03:24:47.422879 24068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:24:47.425113 24068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:24:47.426406 24068 hybrid_clock.cc:648] HybridClock initialized: now 1790220287426343 us; error 60 us; skew 500 ppm
I20260924 03:24:47.426913 24068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:24:47 dist-test-slave-00qs krb5kdc[24055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220287, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:24:47.437232 24068 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:24:47.439539 24068 webserver.cc:460] Webserver started at http://127.10.12.190:43433/ using document root <none> and password file <none>
I20260924 03:24:47.440598 24068 fs_manager.cc:359] Metadata directory not provided
I20260924 03:24:47.440773 24068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:24:47.441166 24068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:24:47.445453 24068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "2c27d25d396544958f11959d54fc32ac"
format_stamp: "Formatted at 2026-09-24 03:24:47 on dist-test-slave-00qs"
server_key: "169856988ebc8c0e649a484ae0f8e2f6"
server_key_iv: "f1e0785976acf21462adcd692bee4f46"
server_key_version: "encryptionkey@0"
I20260924 03:24:47.446871 24068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "2c27d25d396544958f11959d54fc32ac"
format_stamp: "Formatted at 2026-09-24 03:24:47 on dist-test-slave-00qs"
server_key: "169856988ebc8c0e649a484ae0f8e2f6"
server_key_iv: "f1e0785976acf21462adcd692bee4f46"
server_key_version: "encryptionkey@0"
I20260924 03:24:47.454900 24068 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260924 03:24:47.460940 24083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:47.463209 24068 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:24:47.463491 24068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "2c27d25d396544958f11959d54fc32ac"
format_stamp: "Formatted at 2026-09-24 03:24:47 on dist-test-slave-00qs"
server_key: "169856988ebc8c0e649a484ae0f8e2f6"
server_key_iv: "f1e0785976acf21462adcd692bee4f46"
server_key_version: "encryptionkey@0"
I20260924 03:24:47.463801 24068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:24:47.491748 24068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:24:47.498200 24068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:24:47.498613 24068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:24:47.523128 24068 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:37799
I20260924 03:24:47.523161 24134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:37799 every 8 connection(s)
I20260924 03:24:47.525297 24068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:24:47.530485 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 24068
I20260924 03:24:47.530807 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:24:47.531915 10290 external_mini_cluster.cc:1428] Setting key 3cb27cb2a496a6244eb06260cad2c8dc
I20260924 03:24:47.533102 24135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 03:24:47 dist-test-slave-00qs krb5kdc[24055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:24:47.553445 24135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c27d25d396544958f11959d54fc32ac: Bootstrap starting.
I20260924 03:24:47.558161 24135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c27d25d396544958f11959d54fc32ac: Neither blocks nor log segments found. Creating new log.
I20260924 03:24:47.559944 24135 log.cc:824] T 00000000000000000000000000000000 P 2c27d25d396544958f11959d54fc32ac: Log is configured to *not* fsync() on all Append() calls
I20260924 03:24:47.564713 24135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c27d25d396544958f11959d54fc32ac: No bootstrap required, opened a new log
I20260924 03:24:47.565160 24138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:47.534610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44830 (local address 127.10.12.190:37799)
0924 03:24:47.535768 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:47.535784 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:47.535905 (+ 121us) server_negotiation.cc:407] Connection header received
0924 03:24:47.537898 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:47.537932 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:47.538573 (+ 641us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:47.539083 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:47.541618 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:47.543623 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:47.546024 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:47.547427 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:47.547680 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:47.554376 (+ 6696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:47.554391 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:47.554416 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:47.554489 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:47.558459 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:47.559888 (+ 1429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:47.559898 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:47.559908 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:47.560320 (+ 412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:47.560935 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:47.560945 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:47.560949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:47.562024 (+ 1075us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:47.562343 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:47.563060 (+ 717us) server_negotiation.cc:299] Negotiation successful
0924 03:24:47.563654 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":440,"threads_started":1}
I20260924 03:24:47.571725 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:24:47.579339 24135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c27d25d396544958f11959d54fc32ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c27d25d396544958f11959d54fc32ac" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37799 } }
I20260924 03:24:47.580205 24135 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c27d25d396544958f11959d54fc32ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:24:47.580379 24135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c27d25d396544958f11959d54fc32ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c27d25d396544958f11959d54fc32ac, State: Initialized, Role: FOLLOWER
I20260924 03:24:47.581208 24135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c27d25d396544958f11959d54fc32ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c27d25d396544958f11959d54fc32ac" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37799 } }
I20260924 03:24:47.581537 24135 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c27d25d396544958f11959d54fc32ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:24:47.581679 24135 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c27d25d396544958f11959d54fc32ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:24:47.581861 24135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c27d25d396544958f11959d54fc32ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:24:47.584820 24135 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c27d25d396544958f11959d54fc32ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c27d25d396544958f11959d54fc32ac" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37799 } }
I20260924 03:24:47.585738 24135 leader_election.cc:302] T 00000000000000000000000000000000 P 2c27d25d396544958f11959d54fc32ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c27d25d396544958f11959d54fc32ac; no voters:
I20260924 03:24:47.586697 24135 leader_election.cc:288] T 00000000000000000000000000000000 P 2c27d25d396544958f11959d54fc32ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:24:47.586939 24140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c27d25d396544958f11959d54fc32ac [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:24:47.587755 24140 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c27d25d396544958f11959d54fc32ac [term 1 LEADER]: Becoming Leader. State: Replica: 2c27d25d396544958f11959d54fc32ac, State: Running, Role: LEADER
I20260924 03:24:47.588528 24140 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c27d25d396544958f11959d54fc32ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c27d25d396544958f11959d54fc32ac" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37799 } }
I20260924 03:24:47.589538 24135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c27d25d396544958f11959d54fc32ac [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:24:47.593070 24141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c27d25d396544958f11959d54fc32ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c27d25d396544958f11959d54fc32ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c27d25d396544958f11959d54fc32ac" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37799 } } }
I20260924 03:24:47.592901 24142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c27d25d396544958f11959d54fc32ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c27d25d396544958f11959d54fc32ac. Latest consensus state: current_term: 1 leader_uuid: "2c27d25d396544958f11959d54fc32ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c27d25d396544958f11959d54fc32ac" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 37799 } } }
I20260924 03:24:47.594079 24141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c27d25d396544958f11959d54fc32ac [sys.catalog]: This master's current role is: LEADER
I20260924 03:24:47.594411 24142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c27d25d396544958f11959d54fc32ac [sys.catalog]: This master's current role is: LEADER
I20260924 03:24:47.596839 24149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:24:47.603049 24149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:24:47.615715 24149 catalog_manager.cc:1347] Generated new cluster ID: f87cc88b8ece4d05a9522dd838c53d6c
I20260924 03:24:47.615886 24149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:24:47.644835 24149 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:24:47.647315 24149 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:24:47.663872 24149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c27d25d396544958f11959d54fc32ac: Generated new TSK 0
I20260924 03:24:47.665889 24149 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:24:47.693933 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 24068
I20260924 03:24:47.695868 24130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:24:47.807539 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 24068
2026-09-24T03:24:47Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2261 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:24:47 dist-test-slave-00qs krb5kdc[24163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:24:47 dist-test-slave-00qs krb5kdc[24163](info): set up 2 sockets
Sep 24 03:24:47 dist-test-slave-00qs krb5kdc[24163](info): commencing operation
krb5kdc: starting...
W20260924 03:24:48.897223 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.001s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:24:49 dist-test-slave-00qs krb5kdc[24163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:24:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:24:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:24:49.197072 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:40883
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:40883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:24:49.635074 24176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:24:49.635504 24176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:24:49.635643 24176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:24:49.644204 24176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:24:49.644534 24176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:24:49.644629 24176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:24:49.644754 24176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:24:49.644816 24176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:24:49.656975 24176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:40883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:40883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:24:49.658947 24176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:24:49.660668 24176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:24:49.675184 24182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:49.675143 24181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:49.675217 24184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:49.675992 24176 server_base.cc:1034] running on GCE node
I20260924 03:24:49.677079 24176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:24:49.679106 24176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:24:49.680390 24176 hybrid_clock.cc:648] HybridClock initialized: now 1790220289680309 us; error 62 us; skew 500 ppm
I20260924 03:24:49.681049 24176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:24:49 dist-test-slave-00qs krb5kdc[24163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220289, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:24:49.685494 24176 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:24:49.687769 24176 webserver.cc:460] Webserver started at http://127.10.12.190:41743/ using document root <none> and password file <none>
I20260924 03:24:49.689040 24176 fs_manager.cc:359] Metadata directory not provided
I20260924 03:24:49.689251 24176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:24:49.689805 24176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:24:49.694355 24176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "b5cdad64e69645e6b5135c0b65000576"
format_stamp: "Formatted at 2026-09-24 03:24:49 on dist-test-slave-00qs"
server_key: "5da1f558ac4c85fc7acb63cc4a61d6ef"
server_key_iv: "ba0eb310dde3e7264e2bb8b058137d77"
server_key_version: "encryptionkey@0"
I20260924 03:24:49.695557 24176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "b5cdad64e69645e6b5135c0b65000576"
format_stamp: "Formatted at 2026-09-24 03:24:49 on dist-test-slave-00qs"
server_key: "5da1f558ac4c85fc7acb63cc4a61d6ef"
server_key_iv: "ba0eb310dde3e7264e2bb8b058137d77"
server_key_version: "encryptionkey@0"
I20260924 03:24:49.703378 24176 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260924 03:24:49.708961 24191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:49.711007 24176 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260924 03:24:49.711244 24176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "b5cdad64e69645e6b5135c0b65000576"
format_stamp: "Formatted at 2026-09-24 03:24:49 on dist-test-slave-00qs"
server_key: "5da1f558ac4c85fc7acb63cc4a61d6ef"
server_key_iv: "ba0eb310dde3e7264e2bb8b058137d77"
server_key_version: "encryptionkey@0"
I20260924 03:24:49.711513 24176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:24:49.730733 24176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:24:49.737192 24176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:24:49.737571 24176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:24:49.760301 24176 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:40883
I20260924 03:24:49.760339 24242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:40883 every 8 connection(s)
I20260924 03:24:49.762187 24176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:24:49.767529 24243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:24:49.770251 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 24176
I20260924 03:24:49.770576 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:24:49.771538 10290 external_mini_cluster.cc:1428] Setting key 778bdf728666afd650e149e6604bfcc5
I20260924 03:24:49.783368 24243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576: Bootstrap starting.
I20260924 03:24:49.787451 24243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576: Neither blocks nor log segments found. Creating new log.
I20260924 03:24:49.788820 24243 log.cc:824] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576: Log is configured to *not* fsync() on all Append() calls
Sep 24 03:24:49 dist-test-slave-00qs krb5kdc[24163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:24:49.793812 24243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576: No bootstrap required, opened a new log
I20260924 03:24:49.802042 24246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:49.774574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34720 (local address 127.10.12.190:40883)
0924 03:24:49.775695 (+ 1121us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:49.775718 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:49.775776 (+ 58us) server_negotiation.cc:407] Connection header received
0924 03:24:49.777873 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:49.777906 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:49.778459 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:49.778933 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:49.780904 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:49.782472 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:49.784032 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:49.784822 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:49.785024 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:49.790576 (+ 5552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:49.790587 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:49.790692 (+ 105us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:49.790765 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:49.795393 (+ 4628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:49.796588 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:49.796599 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:49.796621 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:49.796912 (+ 291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:49.797446 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:49.797452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:49.797456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:49.798464 (+ 1008us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:49.798852 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:49.799372 (+ 520us) server_negotiation.cc:299] Negotiation successful
0924 03:24:49.800075 (+ 703us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":513,"threads_started":1}
I20260924 03:24:49.805617 24243 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5cdad64e69645e6b5135c0b65000576" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 40883 } }
I20260924 03:24:49.806201 24243 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:24:49.806339 24243 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5cdad64e69645e6b5135c0b65000576, State: Initialized, Role: FOLLOWER
I20260924 03:24:49.807113 24243 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5cdad64e69645e6b5135c0b65000576" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 40883 } }
I20260924 03:24:49.807401 24243 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:24:49.807557 24243 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:24:49.807739 24243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:24:49.810657 24243 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5cdad64e69645e6b5135c0b65000576" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 40883 } }
I20260924 03:24:49.811532 24243 leader_election.cc:302] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5cdad64e69645e6b5135c0b65000576; no voters:
I20260924 03:24:49.812484 24243 leader_election.cc:288] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:24:49.812623 24248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:24:49.813448 24248 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [term 1 LEADER]: Becoming Leader. State: Replica: b5cdad64e69645e6b5135c0b65000576, State: Running, Role: LEADER
I20260924 03:24:49.814149 24248 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5cdad64e69645e6b5135c0b65000576" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 40883 } }
I20260924 03:24:49.815217 24243 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:24:49.818758 24249 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5cdad64e69645e6b5135c0b65000576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5cdad64e69645e6b5135c0b65000576" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 40883 } } }
I20260924 03:24:49.818661 24250 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5cdad64e69645e6b5135c0b65000576. Latest consensus state: current_term: 1 leader_uuid: "b5cdad64e69645e6b5135c0b65000576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5cdad64e69645e6b5135c0b65000576" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 40883 } } }
I20260924 03:24:49.819602 24249 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [sys.catalog]: This master's current role is: LEADER
I20260924 03:24:49.819700 24250 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [sys.catalog]: This master's current role is: LEADER
I20260924 03:24:49.821801 24254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:24:49.829058 24254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:24:49.843152 24254 catalog_manager.cc:1347] Generated new cluster ID: 7dc26a5347be49d39de2eb2e2bac1a7d
I20260924 03:24:49.843319 24254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:24:49.856756 24254 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:24:49.858527 24254 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:24:49.872535 24254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576: Generated new TSK 0
I20260924 03:24:49.874071 24254 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
I20260924 03:24:49.948462 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:40883
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:24:50.359310 24269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:24:50.359642 24269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:24:50.359771 24269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:24:50.367260 24269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:24:50.367412 24269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:24:50.367571 24269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:24:50.378463 24269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:40883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:24:50.380647 24269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:24:50.382303 24269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:24:50.398722 24274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:50.398747 24275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:50.399324 24277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:50.399993 24269 server_base.cc:1034] running on GCE node
I20260924 03:24:50.400699 24269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:24:50.401849 24269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:24:50.403163 24269 hybrid_clock.cc:648] HybridClock initialized: now 1790220290403105 us; error 50 us; skew 500 ppm
I20260924 03:24:50.403576 24269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:24:50 dist-test-slave-00qs krb5kdc[24163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220290, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:24:50.407369 24269 init.cc:377] Logged in from keytab as kudu/127.10.12.129@KRBTEST.COM (short username kudu)
I20260924 03:24:50.409574 24269 webserver.cc:460] Webserver started at http://127.10.12.129:38227/ using document root <none> and password file <none>
I20260924 03:24:50.410722 24269 fs_manager.cc:359] Metadata directory not provided
I20260924 03:24:50.410871 24269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:24:50.411181 24269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:24:50.414762 24269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "1f7843376981417a9629a0124f527e25"
format_stamp: "Formatted at 2026-09-24 03:24:50 on dist-test-slave-00qs"
server_key: "c99f92db7eac903e5eb01c9fa3ea69a8"
server_key_iv: "76a425d8b9cb072ecc2ec7919447475a"
server_key_version: "encryptionkey@0"
I20260924 03:24:50.415776 24269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "1f7843376981417a9629a0124f527e25"
format_stamp: "Formatted at 2026-09-24 03:24:50 on dist-test-slave-00qs"
server_key: "c99f92db7eac903e5eb01c9fa3ea69a8"
server_key_iv: "76a425d8b9cb072ecc2ec7919447475a"
server_key_version: "encryptionkey@0"
I20260924 03:24:50.422551 24269 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 03:24:50.427096 24284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:50.428892 24269 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:24:50.429095 24269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "1f7843376981417a9629a0124f527e25"
format_stamp: "Formatted at 2026-09-24 03:24:50 on dist-test-slave-00qs"
server_key: "c99f92db7eac903e5eb01c9fa3ea69a8"
server_key_iv: "76a425d8b9cb072ecc2ec7919447475a"
server_key_version: "encryptionkey@0"
I20260924 03:24:50.429316 24269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:24:50.462778 24269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:24:50.469403 24269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:24:50.470626 24269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:24:50.472399 24269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:24:50.474575 24269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:24:50.474704 24269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:50.474849 24269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:24:50.474915 24269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:50.511173 24269 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:35751
I20260924 03:24:50.511214 24396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:35751 every 8 connection(s)
I20260924 03:24:50.512666 24269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:24:50.514137 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 24269
I20260924 03:24:50.514339 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:24:50.515015 10290 external_mini_cluster.cc:1428] Setting key e3b5b8f15486ba14749a36b589c04382
Sep 24 03:24:50 dist-test-slave-00qs krb5kdc[24163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220290, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.12.130@KRBTEST.COM; defaulting to no policy
I20260924 03:24:50.540771 24399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:50.515579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:49051 (local address 127.10.12.190:40883)
0924 03:24:50.516378 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:50.516385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:50.517772 (+ 1387us) server_negotiation.cc:407] Connection header received
0924 03:24:50.520191 (+ 2419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:50.520197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:50.520298 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:50.520394 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:50.522578 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:50.523624 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:50.526055 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:50.526645 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:50.526773 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:50.532538 (+ 5765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:50.532545 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:50.532550 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:50.532603 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:50.536338 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:50.537588 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:50.537595 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:50.537598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:50.537979 (+ 381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:50.538715 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:50.538720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:50.538723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:50.539049 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:50.539206 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:50.540109 (+ 903us) server_negotiation.cc:299] Negotiation successful
0924 03:24:50.540408 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":502,"threads_started":1}
I20260924 03:24:50.541920 24400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:50.516283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.129:49051)
0924 03:24:50.517368 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:50.517441 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:50.519760 (+ 2319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:50.520698 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:50.520711 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:50.521464 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:24:50.522320 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:50.522348 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:50.523860 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:50.523866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:50.525831 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:50.525852 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:50.527122 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:50.527129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:50.527442 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:50.528347 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:24:50.528385 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:24:50.532136 (+ 3751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:24:50.536621 (+ 4485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:50.536634 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:50.536655 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:50.537384 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:50.538233 (+ 849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:50.538239 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:50.538241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:50.538549 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:50.539195 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:24:50.539205 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:24:50.539611 (+ 406us) client_negotiation.cc:769] Sending connection context
0924 03:24:50.539909 (+ 298us) client_negotiation.cc:240] Negotiation successful
0924 03:24:50.540539 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":550,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.130@KRBTEST.COM" created.
I20260924 03:24:50.544443 24397 heartbeater.cc:347] Connected to a master server at 127.10.12.190:40883
I20260924 03:24:50.544857 24397 heartbeater.cc:464] Registering TS with master...
I20260924 03:24:50.545861 24397 heartbeater.cc:511] Master 127.10.12.190:40883 requested a full tablet report, sending...
I20260924 03:24:50.548810 24208 ts_manager.cc:194] Registered new tserver with Master: 1f7843376981417a9629a0124f527e25 (127.10.12.129:35751)
I20260924 03:24:50.550705 24208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.129@KRBTEST.COM'} at 127.10.12.129:49051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
I20260924 03:24:50.572353 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:40883
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:24:50.927901 24403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:24:50.928258 24403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:24:50.928390 24403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:24:50.936457 24403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:24:50.936597 24403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:24:50.936764 24403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:24:50.947189 24403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:40883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:24:50.948565 24403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:24:50.949884 24403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:24:50.963617 24409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:50.963647 24411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:50.963688 24408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:50.964623 24403 server_base.cc:1034] running on GCE node
I20260924 03:24:50.965309 24403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:24:50.966410 24403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:24:50.967643 24403 hybrid_clock.cc:648] HybridClock initialized: now 1790220290967583 us; error 49 us; skew 500 ppm
I20260924 03:24:50.968010 24403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:24:50 dist-test-slave-00qs krb5kdc[24163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220290, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:24:50.971436 24403 init.cc:377] Logged in from keytab as kudu/127.10.12.130@KRBTEST.COM (short username kudu)
I20260924 03:24:50.973217 24403 webserver.cc:460] Webserver started at http://127.10.12.130:46655/ using document root <none> and password file <none>
I20260924 03:24:50.974014 24403 fs_manager.cc:359] Metadata directory not provided
I20260924 03:24:50.974128 24403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:24:50.974419 24403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:24:50.977499 24403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "fd88482e1b8d4a728220a423d383d7c0"
format_stamp: "Formatted at 2026-09-24 03:24:50 on dist-test-slave-00qs"
server_key: "59a20f0fee691e34fa98d0e768f4a1fa"
server_key_iv: "df6e565cef00b8fc2f9654c2727452b6"
server_key_version: "encryptionkey@0"
I20260924 03:24:50.978307 24403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "fd88482e1b8d4a728220a423d383d7c0"
format_stamp: "Formatted at 2026-09-24 03:24:50 on dist-test-slave-00qs"
server_key: "59a20f0fee691e34fa98d0e768f4a1fa"
server_key_iv: "df6e565cef00b8fc2f9654c2727452b6"
server_key_version: "encryptionkey@0"
I20260924 03:24:50.983904 24403 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 03:24:50.987748 24418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:50.989418 24403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:24:50.989593 24403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "fd88482e1b8d4a728220a423d383d7c0"
format_stamp: "Formatted at 2026-09-24 03:24:50 on dist-test-slave-00qs"
server_key: "59a20f0fee691e34fa98d0e768f4a1fa"
server_key_iv: "df6e565cef00b8fc2f9654c2727452b6"
server_key_version: "encryptionkey@0"
I20260924 03:24:50.989794 24403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:24:51.031561 24403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:24:51.036578 24403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:24:51.036875 24403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:24:51.037881 24403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:24:51.039325 24403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:24:51.039402 24403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:51.039503 24403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:24:51.039569 24403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:51.072770 24403 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:41635
I20260924 03:24:51.072839 24530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:41635 every 8 connection(s)
I20260924 03:24:51.074087 24403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:24:51.077193 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 24403
I20260924 03:24:51.077365 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:24:51.077783 10290 external_mini_cluster.cc:1428] Setting key 73882525c443341ed0b2facd42de8bd0
Sep 24 03:24:51 dist-test-slave-00qs krb5kdc[24163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220290, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:24:51.097208 24533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:51.076670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:59023 (local address 127.10.12.190:40883)
0924 03:24:51.077180 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:51.077187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:51.078207 (+ 1020us) server_negotiation.cc:407] Connection header received
0924 03:24:51.080450 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:51.080455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:51.080613 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:51.080739 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:51.082441 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.083453 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.085254 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.085849 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.085993 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:51.090492 (+ 4499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:51.090498 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:51.090502 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:51.090549 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:51.093340 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:51.094416 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:51.094421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:51.094423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:51.094608 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:51.095166 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:51.095170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:51.095172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:51.095478 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:51.095669 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:51.096674 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0924 03:24:51.096921 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":251,"threads_started":1}
I20260924 03:24:51.098639 24534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:51.077191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.130:59023)
0924 03:24:51.078013 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:51.078068 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:51.080143 (+ 2075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:51.081015 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:51.081025 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:51.081594 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:24:51.082203 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:51.082223 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.083655 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.083659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:51.085069 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:51.085085 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.086121 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.086127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:51.086347 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:51.087190 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:24:51.087216 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:24:51.090128 (+ 2912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:24:51.093548 (+ 3420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:51.093558 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:51.093597 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:51.094258 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:51.094735 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:51.094739 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:51.094742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:51.095045 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:51.095675 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:24:51.095683 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:24:51.096198 (+ 515us) client_negotiation.cc:769] Sending connection context
0924 03:24:51.096526 (+ 328us) client_negotiation.cc:240] Negotiation successful
0924 03:24:51.097109 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":346,"threads_started":1}
I20260924 03:24:51.100423 24531 heartbeater.cc:347] Connected to a master server at 127.10.12.190:40883
I20260924 03:24:51.100754 24531 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.12.131@KRBTEST.COM; defaulting to no policy
I20260924 03:24:51.101694 24531 heartbeater.cc:511] Master 127.10.12.190:40883 requested a full tablet report, sending...
I20260924 03:24:51.103354 24208 ts_manager.cc:194] Registered new tserver with Master: fd88482e1b8d4a728220a423d383d7c0 (127.10.12.130:41635)
I20260924 03:24:51.104292 24208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.130@KRBTEST.COM'} at 127.10.12.130:59023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
I20260924 03:24:51.136438 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:40883
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:24:51.542052 24537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:24:51.542461 24537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:24:51.542603 24537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:24:51.550680 24537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:24:51.550851 24537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:24:51.551025 24537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:24:51.554297 24397 heartbeater.cc:503] Master 127.10.12.190:40883 was elected leader, sending a full tablet report...
I20260924 03:24:51.563009 24537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:40883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:24:51.564873 24537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:24:51.566545 24537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:24:51.582976 24542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:51.582883 24543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:51.583472 24545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:51.583914 24537 server_base.cc:1034] running on GCE node
I20260924 03:24:51.584844 24537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:24:51.586530 24537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:24:51.587934 24537 hybrid_clock.cc:648] HybridClock initialized: now 1790220291587798 us; error 127 us; skew 500 ppm
I20260924 03:24:51.588507 24537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:24:51 dist-test-slave-00qs krb5kdc[24163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220291, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:24:51.592546 24537 init.cc:377] Logged in from keytab as kudu/127.10.12.131@KRBTEST.COM (short username kudu)
I20260924 03:24:51.594600 24537 webserver.cc:460] Webserver started at http://127.10.12.131:34165/ using document root <none> and password file <none>
I20260924 03:24:51.595582 24537 fs_manager.cc:359] Metadata directory not provided
I20260924 03:24:51.595731 24537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:24:51.596171 24537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:24:51.600219 24537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "0559b9773b154613b98b7bd99a46f987"
format_stamp: "Formatted at 2026-09-24 03:24:51 on dist-test-slave-00qs"
server_key: "52c8c77eab9473ff0112f51bdadf04e8"
server_key_iv: "aade1740265a79dccd63b99042f3de9b"
server_key_version: "encryptionkey@0"
I20260924 03:24:51.601297 24537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "0559b9773b154613b98b7bd99a46f987"
format_stamp: "Formatted at 2026-09-24 03:24:51 on dist-test-slave-00qs"
server_key: "52c8c77eab9473ff0112f51bdadf04e8"
server_key_iv: "aade1740265a79dccd63b99042f3de9b"
server_key_version: "encryptionkey@0"
I20260924 03:24:51.608318 24537 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260924 03:24:51.612757 24552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:51.614471 24537 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:24:51.614657 24537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "0559b9773b154613b98b7bd99a46f987"
format_stamp: "Formatted at 2026-09-24 03:24:51 on dist-test-slave-00qs"
server_key: "52c8c77eab9473ff0112f51bdadf04e8"
server_key_iv: "aade1740265a79dccd63b99042f3de9b"
server_key_version: "encryptionkey@0"
I20260924 03:24:51.614897 24537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:24:51.640763 24537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:24:51.646296 24537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:24:51.646641 24537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:24:51.648043 24537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:24:51.649873 24537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:24:51.649976 24537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:51.650105 24537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:24:51.650169 24537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:51.687891 24537 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:40173
I20260924 03:24:51.687930 24664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:40173 every 8 connection(s)
I20260924 03:24:51.690132 24537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:24:51.694695 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 24537
I20260924 03:24:51.694941 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:24:51.695643 10290 external_mini_cluster.cc:1428] Setting key 78e2ed5481be59d52b38df31f0f52ec2
Sep 24 03:24:51 dist-test-slave-00qs krb5kdc[24163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220291, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:24:51.718624 24667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:51.694230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:34625 (local address 127.10.12.190:40883)
0924 03:24:51.694938 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:51.694946 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:51.696315 (+ 1369us) server_negotiation.cc:407] Connection header received
0924 03:24:51.698969 (+ 2654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:51.698976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:51.699093 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:51.699210 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:51.701698 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.702670 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.704865 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.705363 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.705505 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:51.710648 (+ 5143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:51.710656 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:51.710660 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:51.710716 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:51.714155 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:51.715355 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:51.715362 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:51.715365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:51.715553 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:51.716387 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:51.716392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:51.716395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:51.716838 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:51.717006 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:51.717975 (+ 969us) server_negotiation.cc:299] Negotiation successful
0924 03:24:51.718245 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":336,"threads_started":1}
I20260924 03:24:51.719933 24668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:51.694874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.131:34625)
0924 03:24:51.696031 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:51.696108 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:51.698537 (+ 2429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:51.699573 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:51.699585 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:51.700472 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:24:51.701397 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:51.701427 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.702893 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.702898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:51.704612 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:51.704632 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.705685 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.705694 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:51.706023 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:51.706911 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:24:51.706943 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:24:51.710096 (+ 3153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:24:51.714368 (+ 4272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:51.714379 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:51.714399 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:51.715160 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:51.715813 (+ 653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:51.715818 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:51.715821 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:51.716241 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:51.717008 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:24:51.717017 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:24:51.717517 (+ 500us) client_negotiation.cc:769] Sending connection context
0924 03:24:51.717834 (+ 317us) client_negotiation.cc:240] Negotiation successful
0924 03:24:51.718412 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":395,"threads_started":1}
I20260924 03:24:51.722340 24665 heartbeater.cc:347] Connected to a master server at 127.10.12.190:40883
I20260924 03:24:51.722795 24665 heartbeater.cc:464] Registering TS with master...
I20260924 03:24:51.723831 24665 heartbeater.cc:511] Master 127.10.12.190:40883 requested a full tablet report, sending...
I20260924 03:24:51.725936 24208 ts_manager.cc:194] Registered new tserver with Master: 0559b9773b154613b98b7bd99a46f987 (127.10.12.131:40173)
I20260924 03:24:51.727111 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:24:51.727176 24208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.131@KRBTEST.COM'} at 127.10.12.131:34625
I20260924 03:24:51.751588 24667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:51.734038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.10.12.190:40883)
0924 03:24:51.734416 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:51.734425 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:51.734748 (+ 323us) server_negotiation.cc:407] Connection header received
0924 03:24:51.735022 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:51.735028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:51.735119 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:51.735219 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:51.737214 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.738109 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.739728 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.740292 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.740443 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:51.743773 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:51.743782 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:51.743785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:51.743846 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:51.747289 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:51.748481 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:51.748486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:51.748489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:51.748676 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:51.749232 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:51.749238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:51.749306 (+ 68us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:51.749922 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:51.750262 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:51.750786 (+ 524us) server_negotiation.cc:299] Negotiation successful
0924 03:24:51.751112 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260924 03:24:51.758065 24208 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:24:51.767803 24208 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:24:51.799230 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:51.785990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:40173 (local address 127.0.0.1:46590)
0924 03:24:51.786870 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:51.786918 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:51.787142 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:51.788404 (+ 1262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:51.788413 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:51.788450 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:24:51.788981 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:51.788997 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.791403 (+ 2406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.791411 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:51.794585 (+ 3174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:51.794620 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.797692 (+ 3072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.797701 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:51.798014 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:51.798105 (+ 91us) client_negotiation.cc:769] Sending connection context
0924 03:24:51.798311 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 03:24:51.798631 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":338,"threads_started":1}
I20260924 03:24:51.800555 24679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:51.786232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46590 (local address 127.10.12.131:40173)
0924 03:24:51.787135 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:51.787145 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:51.787190 (+ 45us) server_negotiation.cc:407] Connection header received
0924 03:24:51.787415 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:51.787424 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:51.787763 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:51.788018 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:24:51.789256 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.791134 (+ 1878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.794948 (+ 3814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.796918 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.797133 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:51.797471 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:51.799849 (+ 2378us) server_negotiation.cc:299] Negotiation successful
0924 03:24:51.800169 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":1710,"threads_started":1}
I20260924 03:24:51.806346 24600 tablet_service.cc:1467] Processing CreateTablet for tablet 0b0accbfa7864d849bb008660c6ddd5f (DEFAULT_TABLE table=test-table [id=4df956d1243c403fbb1deecf89ae127b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:24:51.807986 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:51.787732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35000 (local address 127.10.12.129:35751)
0924 03:24:51.796994 (+ 9262us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:51.797009 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:51.797744 (+ 735us) server_negotiation.cc:407] Connection header received
0924 03:24:51.798010 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:51.798024 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:51.798536 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:51.798885 (+ 349us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:24:51.800528 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.802228 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.804972 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.806704 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.806873 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:51.807174 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:51.807385 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 03:24:51.807600 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8958,"thread_start_us":8709,"threads_started":1}
I20260924 03:24:51.807981 24680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:51.786976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:35751 (local address 127.0.0.1:35000)
0924 03:24:51.797571 (+ 10595us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:51.797596 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:51.797800 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:51.799744 (+ 1944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:51.799749 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:51.799783 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:24:51.800250 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:51.800267 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.802425 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.802431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:51.804716 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:51.804732 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.806944 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.806952 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:51.807146 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:51.807183 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 03:24:51.807290 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 03:24:51.807511 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10418,"thread_start_us":10231,"threads_started":1}
I20260924 03:24:51.809024 24600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b0accbfa7864d849bb008660c6ddd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:24:51.810063 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:51.788463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:41635 (local address 127.0.0.1:55488)
0924 03:24:51.800316 (+ 11853us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:51.800338 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:51.800455 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:51.801801 (+ 1346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:51.801808 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:51.801841 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:24:51.802116 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:51.802126 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.804346 (+ 2220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.804353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:51.806586 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:51.806607 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.808715 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.808723 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:51.808928 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:51.808968 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 03:24:51.809510 (+ 542us) client_negotiation.cc:240] Negotiation successful
0924 03:24:51.809719 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11739,"thread_start_us":16565,"threads_started":1}
I20260924 03:24:51.811525 24683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:51.788830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55488 (local address 127.10.12.130:41635)
0924 03:24:51.799631 (+ 10801us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:51.799638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:51.800590 (+ 952us) server_negotiation.cc:407] Connection header received
0924 03:24:51.800769 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:51.800779 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:51.801142 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:51.801417 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:24:51.802329 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.804080 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.806793 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.808489 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.808672 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:51.808910 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:51.810650 (+ 1740us) server_negotiation.cc:299] Negotiation successful
0924 03:24:51.810931 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10606,"thread_start_us":10418,"threads_started":1}
I20260924 03:24:51.814067 24332 tablet_service.cc:1467] Processing CreateTablet for tablet 0b0accbfa7864d849bb008660c6ddd5f (DEFAULT_TABLE table=test-table [id=4df956d1243c403fbb1deecf89ae127b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:24:51.816910 24332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b0accbfa7864d849bb008660c6ddd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:24:51.817451 24466 tablet_service.cc:1467] Processing CreateTablet for tablet 0b0accbfa7864d849bb008660c6ddd5f (DEFAULT_TABLE table=test-table [id=4df956d1243c403fbb1deecf89ae127b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:24:51.819599 24466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b0accbfa7864d849bb008660c6ddd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:24:51.826036 24684 tablet_bootstrap.cc:492] T 0b0accbfa7864d849bb008660c6ddd5f P 0559b9773b154613b98b7bd99a46f987: Bootstrap starting.
I20260924 03:24:51.831139 24684 tablet_bootstrap.cc:654] T 0b0accbfa7864d849bb008660c6ddd5f P 0559b9773b154613b98b7bd99a46f987: Neither blocks nor log segments found. Creating new log.
I20260924 03:24:51.832592 24684 log.cc:824] T 0b0accbfa7864d849bb008660c6ddd5f P 0559b9773b154613b98b7bd99a46f987: Log is configured to *not* fsync() on all Append() calls
I20260924 03:24:51.837157 24684 tablet_bootstrap.cc:492] T 0b0accbfa7864d849bb008660c6ddd5f P 0559b9773b154613b98b7bd99a46f987: No bootstrap required, opened a new log
I20260924 03:24:51.837749 24684 ts_tablet_manager.cc:1397] T 0b0accbfa7864d849bb008660c6ddd5f P 0559b9773b154613b98b7bd99a46f987: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260924 03:24:51.838150 24687 tablet_bootstrap.cc:492] T 0b0accbfa7864d849bb008660c6ddd5f P fd88482e1b8d4a728220a423d383d7c0: Bootstrap starting.
I20260924 03:24:51.838196 24686 tablet_bootstrap.cc:492] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25: Bootstrap starting.
I20260924 03:24:51.842269 24687 tablet_bootstrap.cc:654] T 0b0accbfa7864d849bb008660c6ddd5f P fd88482e1b8d4a728220a423d383d7c0: Neither blocks nor log segments found. Creating new log.
I20260924 03:24:51.842312 24686 tablet_bootstrap.cc:654] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25: Neither blocks nor log segments found. Creating new log.
I20260924 03:24:51.843657 24687 log.cc:824] T 0b0accbfa7864d849bb008660c6ddd5f P fd88482e1b8d4a728220a423d383d7c0: Log is configured to *not* fsync() on all Append() calls
I20260924 03:24:51.843717 24686 log.cc:824] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25: Log is configured to *not* fsync() on all Append() calls
I20260924 03:24:51.848115 24687 tablet_bootstrap.cc:492] T 0b0accbfa7864d849bb008660c6ddd5f P fd88482e1b8d4a728220a423d383d7c0: No bootstrap required, opened a new log
I20260924 03:24:51.848114 24686 tablet_bootstrap.cc:492] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25: No bootstrap required, opened a new log
I20260924 03:24:51.848511 24687 ts_tablet_manager.cc:1397] T 0b0accbfa7864d849bb008660c6ddd5f P fd88482e1b8d4a728220a423d383d7c0: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260924 03:24:51.848511 24686 ts_tablet_manager.cc:1397] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260924 03:24:51.849615 24684 raft_consensus.cc:348] T 0b0accbfa7864d849bb008660c6ddd5f P 0559b9773b154613b98b7bd99a46f987 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559b9773b154613b98b7bd99a46f987" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40173 } } peers { permanent_uuid: "1f7843376981417a9629a0124f527e25" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35751 } } peers { permanent_uuid: "fd88482e1b8d4a728220a423d383d7c0" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 41635 } }
I20260924 03:24:51.850107 24684 raft_consensus.cc:374] T 0b0accbfa7864d849bb008660c6ddd5f P 0559b9773b154613b98b7bd99a46f987 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:24:51.850255 24684 raft_consensus.cc:729] T 0b0accbfa7864d849bb008660c6ddd5f P 0559b9773b154613b98b7bd99a46f987 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0559b9773b154613b98b7bd99a46f987, State: Initialized, Role: FOLLOWER
I20260924 03:24:51.851033 24684 consensus_queue.cc:260] T 0b0accbfa7864d849bb008660c6ddd5f P 0559b9773b154613b98b7bd99a46f987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559b9773b154613b98b7bd99a46f987" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40173 } } peers { permanent_uuid: "1f7843376981417a9629a0124f527e25" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35751 } } peers { permanent_uuid: "fd88482e1b8d4a728220a423d383d7c0" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 41635 } }
I20260924 03:24:51.852506 24665 heartbeater.cc:503] Master 127.10.12.190:40883 was elected leader, sending a full tablet report...
I20260924 03:24:51.853402 24684 ts_tablet_manager.cc:1428] T 0b0accbfa7864d849bb008660c6ddd5f P 0559b9773b154613b98b7bd99a46f987: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260924 03:24:51.860978 24686 raft_consensus.cc:348] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559b9773b154613b98b7bd99a46f987" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40173 } } peers { permanent_uuid: "1f7843376981417a9629a0124f527e25" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35751 } } peers { permanent_uuid: "fd88482e1b8d4a728220a423d383d7c0" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 41635 } }
I20260924 03:24:51.860978 24687 raft_consensus.cc:348] T 0b0accbfa7864d849bb008660c6ddd5f P fd88482e1b8d4a728220a423d383d7c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559b9773b154613b98b7bd99a46f987" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40173 } } peers { permanent_uuid: "1f7843376981417a9629a0124f527e25" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35751 } } peers { permanent_uuid: "fd88482e1b8d4a728220a423d383d7c0" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 41635 } }
I20260924 03:24:51.861683 24687 raft_consensus.cc:374] T 0b0accbfa7864d849bb008660c6ddd5f P fd88482e1b8d4a728220a423d383d7c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:24:51.861683 24686 raft_consensus.cc:374] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:24:51.861815 24687 raft_consensus.cc:729] T 0b0accbfa7864d849bb008660c6ddd5f P fd88482e1b8d4a728220a423d383d7c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd88482e1b8d4a728220a423d383d7c0, State: Initialized, Role: FOLLOWER
I20260924 03:24:51.861816 24686 raft_consensus.cc:729] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f7843376981417a9629a0124f527e25, State: Initialized, Role: FOLLOWER
I20260924 03:24:51.862705 24687 consensus_queue.cc:260] T 0b0accbfa7864d849bb008660c6ddd5f P fd88482e1b8d4a728220a423d383d7c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559b9773b154613b98b7bd99a46f987" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40173 } } peers { permanent_uuid: "1f7843376981417a9629a0124f527e25" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35751 } } peers { permanent_uuid: "fd88482e1b8d4a728220a423d383d7c0" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 41635 } }
I20260924 03:24:51.862705 24686 consensus_queue.cc:260] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559b9773b154613b98b7bd99a46f987" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40173 } } peers { permanent_uuid: "1f7843376981417a9629a0124f527e25" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35751 } } peers { permanent_uuid: "fd88482e1b8d4a728220a423d383d7c0" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 41635 } }
I20260924 03:24:51.864187 24531 heartbeater.cc:503] Master 127.10.12.190:40883 was elected leader, sending a full tablet report...
I20260924 03:24:51.865252 24686 ts_tablet_manager.cc:1428] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
I20260924 03:24:51.865248 24687 ts_tablet_manager.cc:1428] T 0b0accbfa7864d849bb008660c6ddd5f P fd88482e1b8d4a728220a423d383d7c0: Time spent starting tablet: real 0.017s user 0.015s sys 0.001s
W20260924 03:24:51.942440 24666 tablet.cc:2369] T 0b0accbfa7864d849bb008660c6ddd5f P 0559b9773b154613b98b7bd99a46f987: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:24:51.945326 24691 raft_consensus.cc:482] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:24:51.945712 24691 raft_consensus.cc:504] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559b9773b154613b98b7bd99a46f987" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40173 } } peers { permanent_uuid: "1f7843376981417a9629a0124f527e25" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35751 } } peers { permanent_uuid: "fd88482e1b8d4a728220a423d383d7c0" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 41635 } }
I20260924 03:24:51.948060 24691 leader_election.cc:288] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0559b9773b154613b98b7bd99a46f987 (127.10.12.131:40173), fd88482e1b8d4a728220a423d383d7c0 (127.10.12.130:41635)
I20260924 03:24:51.958657 24679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:51.948715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:47357 (local address 127.10.12.131:40173)
0924 03:24:51.949206 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:51.949213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:51.949590 (+ 377us) server_negotiation.cc:407] Connection header received
0924 03:24:51.949830 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:51.949836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:51.949925 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:51.950016 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:24:51.950766 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.952066 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.955661 (+ 3595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.957146 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.957349 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:51.957449 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:51.958167 (+ 718us) server_negotiation.cc:299] Negotiation successful
0924 03:24:51.958338 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301}
I20260924 03:24:51.958738 24683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:51.948710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:48491 (local address 127.10.12.130:41635)
0924 03:24:51.948978 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:51.948987 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:51.949875 (+ 888us) server_negotiation.cc:407] Connection header received
0924 03:24:51.950065 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:51.950071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:51.950154 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:51.950231 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:24:51.950970 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.952092 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.955711 (+ 3619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.957105 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.957349 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:51.957449 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:51.958161 (+ 712us) server_negotiation.cc:299] Negotiation successful
0924 03:24:51.958346 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 03:24:51.958873 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:51.948749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:41635 (local address 127.10.12.129:48491)
0924 03:24:51.949749 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:51.949768 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:51.949922 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:51.950282 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:51.950287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:51.950323 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:24:51.950766 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:51.950778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.952236 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.952242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:51.955336 (+ 3094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:51.955368 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.957532 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.957538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:51.957892 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:51.957961 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 03:24:51.958086 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 03:24:51.958289 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":228,"thread_start_us":655,"threads_started":1}
I20260924 03:24:51.959023 24693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:51.948369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:40173 (local address 127.10.12.129:47357)
0924 03:24:51.949443 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:51.949464 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:51.949681 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:51.950073 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:51.950076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:51.950097 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:24:51.950592 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:51.950607 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.952236 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.952242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:51.955337 (+ 3095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:51.955368 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:51.957519 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:51.957530 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:51.957897 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:51.957961 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 03:24:51.958086 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 03:24:51.958289 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"mutex_wait_us":400,"thread_start_us":691,"threads_started":1}
I20260924 03:24:51.960386 24486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b0accbfa7864d849bb008660c6ddd5f" candidate_uuid: "1f7843376981417a9629a0124f527e25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd88482e1b8d4a728220a423d383d7c0" is_pre_election: true
I20260924 03:24:51.960400 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b0accbfa7864d849bb008660c6ddd5f" candidate_uuid: "1f7843376981417a9629a0124f527e25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0559b9773b154613b98b7bd99a46f987" is_pre_election: true
I20260924 03:24:51.961009 24486 raft_consensus.cc:2454] T 0b0accbfa7864d849bb008660c6ddd5f P fd88482e1b8d4a728220a423d383d7c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f7843376981417a9629a0124f527e25 in term 0.
I20260924 03:24:51.961005 24620 raft_consensus.cc:2454] T 0b0accbfa7864d849bb008660c6ddd5f P 0559b9773b154613b98b7bd99a46f987 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f7843376981417a9629a0124f527e25 in term 0.
I20260924 03:24:51.961882 24288 leader_election.cc:302] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f7843376981417a9629a0124f527e25, fd88482e1b8d4a728220a423d383d7c0; no voters:
I20260924 03:24:51.962453 24691 raft_consensus.cc:2781] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:24:51.962695 24691 raft_consensus.cc:482] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:24:51.962816 24691 raft_consensus.cc:3037] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:24:51.965651 24691 raft_consensus.cc:504] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559b9773b154613b98b7bd99a46f987" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40173 } } peers { permanent_uuid: "1f7843376981417a9629a0124f527e25" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35751 } } peers { permanent_uuid: "fd88482e1b8d4a728220a423d383d7c0" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 41635 } }
I20260924 03:24:51.966383 24691 leader_election.cc:288] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25 [CANDIDATE]: Term 1 election: Requested vote from peers 0559b9773b154613b98b7bd99a46f987 (127.10.12.131:40173), fd88482e1b8d4a728220a423d383d7c0 (127.10.12.130:41635)
I20260924 03:24:51.967111 24486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b0accbfa7864d849bb008660c6ddd5f" candidate_uuid: "1f7843376981417a9629a0124f527e25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd88482e1b8d4a728220a423d383d7c0"
I20260924 03:24:51.967096 24620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b0accbfa7864d849bb008660c6ddd5f" candidate_uuid: "1f7843376981417a9629a0124f527e25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0559b9773b154613b98b7bd99a46f987"
I20260924 03:24:51.967391 24486 raft_consensus.cc:3037] T 0b0accbfa7864d849bb008660c6ddd5f P fd88482e1b8d4a728220a423d383d7c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:24:51.967384 24620 raft_consensus.cc:3037] T 0b0accbfa7864d849bb008660c6ddd5f P 0559b9773b154613b98b7bd99a46f987 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:24:51.971232 24620 raft_consensus.cc:2454] T 0b0accbfa7864d849bb008660c6ddd5f P 0559b9773b154613b98b7bd99a46f987 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f7843376981417a9629a0124f527e25 in term 1.
I20260924 03:24:51.971232 24486 raft_consensus.cc:2454] T 0b0accbfa7864d849bb008660c6ddd5f P fd88482e1b8d4a728220a423d383d7c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f7843376981417a9629a0124f527e25 in term 1.
I20260924 03:24:51.972025 24288 leader_election.cc:302] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0559b9773b154613b98b7bd99a46f987, 1f7843376981417a9629a0124f527e25, fd88482e1b8d4a728220a423d383d7c0; no voters:
I20260924 03:24:51.972512 24691 raft_consensus.cc:2781] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:24:51.973220 24691 raft_consensus.cc:686] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25 [term 1 LEADER]: Becoming Leader. State: Replica: 1f7843376981417a9629a0124f527e25, State: Running, Role: LEADER
I20260924 03:24:51.973961 24691 consensus_queue.cc:237] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559b9773b154613b98b7bd99a46f987" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40173 } } peers { permanent_uuid: "1f7843376981417a9629a0124f527e25" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35751 } } peers { permanent_uuid: "fd88482e1b8d4a728220a423d383d7c0" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 41635 } }
I20260924 03:24:51.982582 24208 catalog_manager.cc:5495] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f7843376981417a9629a0124f527e25 (127.10.12.129). New cstate: current_term: 1 leader_uuid: "1f7843376981417a9629a0124f527e25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559b9773b154613b98b7bd99a46f987" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f7843376981417a9629a0124f527e25" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd88482e1b8d4a728220a423d383d7c0" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 41635 } health_report { overall_health: UNKNOWN } } }
W20260924 03:24:52.015753 24398 tablet.cc:2369] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:24:52.018759 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:52.011746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35002 (local address 127.10.12.129:35751)
0924 03:24:52.012056 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:52.012061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:52.012084 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:24:52.012318 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:52.012323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:52.012451 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:52.012601 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:24:52.013169 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:52.013971 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:52.015810 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:52.016550 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:52.016666 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:52.017330 (+ 664us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:24:52.018094 (+ 764us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:24:52.018240 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:52.018392 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 03:24:52.018486 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 03:24:52.030838 24620 raft_consensus.cc:1261] T 0b0accbfa7864d849bb008660c6ddd5f P 0559b9773b154613b98b7bd99a46f987 [term 1 FOLLOWER]: Refusing update from remote peer 1f7843376981417a9629a0124f527e25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:24:52.030840 24486 raft_consensus.cc:1261] T 0b0accbfa7864d849bb008660c6ddd5f P fd88482e1b8d4a728220a423d383d7c0 [term 1 FOLLOWER]: Refusing update from remote peer 1f7843376981417a9629a0124f527e25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:24:52.032475 24695 consensus_queue.cc:1035] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd88482e1b8d4a728220a423d383d7c0" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 41635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:24:52.033075 24691 consensus_queue.cc:1035] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0559b9773b154613b98b7bd99a46f987" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:24:52.052512 24699 mvcc.cc:205] Tried to move back new op lower bound from 7332742316142645248 to 7332742315930996736. Current Snapshot: MvccSnapshot[applied={T|T < 7332742316142645248}]
I20260924 03:24:52.057360 24700 mvcc.cc:205] Tried to move back new op lower bound from 7332742316142645248 to 7332742315930996736. Current Snapshot: MvccSnapshot[applied={T|T < 7332742316142645248}]
I20260924 03:24:52.066437 24208 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:24:52.066774 24208 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:24:52.071769 24208 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576: Sending DeleteTablet for 3 replicas of tablet 0b0accbfa7864d849bb008660c6ddd5f
I20260924 03:24:52.074133 24600 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b0accbfa7864d849bb008660c6ddd5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:24:52 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:46590
I20260924 03:24:52.074513 24332 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b0accbfa7864d849bb008660c6ddd5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:24:52 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:35000
I20260924 03:24:52.074824 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 24269
I20260924 03:24:52.075181 24466 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b0accbfa7864d849bb008660c6ddd5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:24:52 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:55488
I20260924 03:24:52.106300 24710 tablet_replica.cc:330] T 0b0accbfa7864d849bb008660c6ddd5f P fd88482e1b8d4a728220a423d383d7c0: stopping tablet replica
I20260924 03:24:52.107065 24710 raft_consensus.cc:2229] T 0b0accbfa7864d849bb008660c6ddd5f P fd88482e1b8d4a728220a423d383d7c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:24:52.112314 24708 tablet_replica.cc:330] T 0b0accbfa7864d849bb008660c6ddd5f P 0559b9773b154613b98b7bd99a46f987: stopping tablet replica
I20260924 03:24:52.107895 24710 raft_consensus.cc:2258] T 0b0accbfa7864d849bb008660c6ddd5f P fd88482e1b8d4a728220a423d383d7c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:24:52.113066 24708 raft_consensus.cc:2229] T 0b0accbfa7864d849bb008660c6ddd5f P 0559b9773b154613b98b7bd99a46f987 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:24:52.113922 24708 raft_consensus.cc:2258] T 0b0accbfa7864d849bb008660c6ddd5f P 0559b9773b154613b98b7bd99a46f987 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:24:52.117851 24708 ts_tablet_manager.cc:1905] T 0b0accbfa7864d849bb008660c6ddd5f P 0559b9773b154613b98b7bd99a46f987: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:24:52.118708 24710 ts_tablet_manager.cc:1905] T 0b0accbfa7864d849bb008660c6ddd5f P fd88482e1b8d4a728220a423d383d7c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:24:52.120183 24709 tablet_replica.cc:330] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25: stopping tablet replica
I20260924 03:24:52.120929 24709 raft_consensus.cc:2229] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:24:52.121968 24709 raft_consensus.cc:2258] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:24:52.127993 24708 ts_tablet_manager.cc:1918] T 0b0accbfa7864d849bb008660c6ddd5f P 0559b9773b154613b98b7bd99a46f987: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 24 03:24:52 dist-test-slave-00qs krb5kdc[24163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.129@KRBTEST.COM
I20260924 03:24:52.128521 24708 log.cc:1196] T 0b0accbfa7864d849bb008660c6ddd5f P 0559b9773b154613b98b7bd99a46f987: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/0b0accbfa7864d849bb008660c6ddd5f
I20260924 03:24:52.129271 24708 ts_tablet_manager.cc:1939] T 0b0accbfa7864d849bb008660c6ddd5f P 0559b9773b154613b98b7bd99a46f987: Deleting consensus metadata
I20260924 03:24:52.130381 24710 ts_tablet_manager.cc:1918] T 0b0accbfa7864d849bb008660c6ddd5f P fd88482e1b8d4a728220a423d383d7c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:24:52.130640 24710 log.cc:1196] T 0b0accbfa7864d849bb008660c6ddd5f P fd88482e1b8d4a728220a423d383d7c0: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/0b0accbfa7864d849bb008660c6ddd5f
I20260924 03:24:52.132201 24192 catalog_manager.cc:4841] TS 0559b9773b154613b98b7bd99a46f987 (127.10.12.131:40173): tablet 0b0accbfa7864d849bb008660c6ddd5f (table test-table [id=4df956d1243c403fbb1deecf89ae127b]) successfully deleted
I20260924 03:24:52.133200 24710 ts_tablet_manager.cc:1939] T 0b0accbfa7864d849bb008660c6ddd5f P fd88482e1b8d4a728220a423d383d7c0: Deleting consensus metadata
I20260924 03:24:52.133796 24709 ts_tablet_manager.cc:1905] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:24:52.136092 24195 catalog_manager.cc:4841] TS fd88482e1b8d4a728220a423d383d7c0 (127.10.12.130:41635): tablet 0b0accbfa7864d849bb008660c6ddd5f (table test-table [id=4df956d1243c403fbb1deecf89ae127b]) successfully deleted
I20260924 03:24:52.139492 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:52.076241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35014 (local address 127.10.12.129:35751)
0924 03:24:52.076458 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:52.076466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:52.105546 (+ 29080us) server_negotiation.cc:407] Connection header received
0924 03:24:52.108611 (+ 3065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:52.108621 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:52.108749 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:52.108847 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:52.113412 (+ 4565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:52.114980 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:52.121796 (+ 6816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:52.122588 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:52.122728 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:52.129875 (+ 7147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:52.129904 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:52.129930 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:52.129997 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:52.134894 (+ 4897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:52.136261 (+ 1367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:52.136275 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:52.136279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:52.136533 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:52.137160 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:52.137166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:52.137169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:52.137842 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:52.138298 (+ 456us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:52.138714 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 03:24:52.139051 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 03:24:52.140462 24392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:24:52.257169 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 24269
I20260924 03:24:52.275708 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 24403
W20260924 03:24:52.275920 24194 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.12.129:35751 (error 108)
I20260924 03:24:52.277988 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:52.277488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:35751 (local address 127.0.0.1:35022)
0924 03:24:52.277686 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:52.277803 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:35751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 03:24:52.278220 24194 catalog_manager.cc:4584] TS 1f7843376981417a9629a0124f527e25 (127.10.12.129:35751): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0b0accbfa7864d849bb008660c6ddd5f: Network error: Client connection negotiation failed: client connection to 127.10.12.129:35751: connect: Connection refused (error 111)
Sep 24 03:24:52 dist-test-slave-00qs krb5kdc[24163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.130@KRBTEST.COM
I20260924 03:24:52.294445 24683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:52.276906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55494 (local address 127.10.12.130:41635)
0924 03:24:52.277116 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:52.277123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:52.277213 (+ 90us) server_negotiation.cc:407] Connection header received
0924 03:24:52.277441 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:52.277446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:52.277523 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:52.277629 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:52.279204 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:52.280117 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:52.281616 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:52.282067 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:52.282193 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:52.287502 (+ 5309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:52.287525 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:52.287546 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:52.287610 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:52.290994 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:52.291901 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:52.291913 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:52.291916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:52.292137 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:52.292537 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:52.292543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:52.292547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:52.293083 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:52.293395 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:52.293777 (+ 382us) server_negotiation.cc:299] Negotiation successful
0924 03:24:52.294030 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 03:24:52.295264 24526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:24:52.332404 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:52.331789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:35751 (local address 127.0.0.1:35034)
0924 03:24:52.332073 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:52.332194 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:35751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 03:24:52.333940 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:52.333375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:35751 (local address 127.0.0.1:35036)
0924 03:24:52.333594 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:52.333768 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:35751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 03:24:52.395205 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:52.394460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:35751 (local address 127.0.0.1:35044)
0924 03:24:52.394838 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:52.394990 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:35751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260924 03:24:52.396850 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:52.396221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:35751 (local address 127.0.0.1:35048)
0924 03:24:52.396505 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:52.396620 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:35751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260924 03:24:52.409924 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 24403
I20260924 03:24:52.428568 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 24537
Sep 24 03:24:52 dist-test-slave-00qs krb5kdc[24163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.131@KRBTEST.COM
I20260924 03:24:52.455142 24679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:52.429682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46606 (local address 127.10.12.131:40173)
0924 03:24:52.429898 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:52.429904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:52.429928 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:24:52.430011 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:52.430017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:52.430107 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:52.430211 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:52.432415 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:52.433430 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:52.435153 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:52.435628 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:52.435770 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:52.446776 (+ 11006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:52.446802 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:52.446824 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:52.446887 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:52.450721 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:52.452068 (+ 1347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:52.452080 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:52.452082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:52.452334 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:52.452904 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:52.452910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:52.452912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:52.453485 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:52.453815 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:52.454308 (+ 493us) server_negotiation.cc:299] Negotiation successful
0924 03:24:52.454750 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:24:52.456308 24660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:24:52.478191 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:52.477425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:35751 (local address 127.0.0.1:35052)
0924 03:24:52.477806 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:52.477961 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:35751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260924 03:24:52.480134 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:52.479338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:35751 (local address 127.0.0.1:35064)
0924 03:24:52.479604 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:52.479828 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:35751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260924 03:24:52.562618 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 24537
I20260924 03:24:52.577862 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 24176
I20260924 03:24:52.579298 24238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:24:52.716984 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:52.716218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:35751 (local address 127.0.0.1:35080)
0924 03:24:52.716602 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:52.716735 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:35751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206}
I20260924 03:24:52.718550 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 24176
I20260924 03:24:52.746165 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:40883
--webserver_interface=127.10.12.190
--webserver_port=41743
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:40883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:24:53.166046 24717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:24:53.166482 24717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:24:53.166621 24717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:24:53.174784 24717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:24:53.174947 24717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:24:53.175022 24717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:24:53.175103 24717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:24:53.175165 24717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:24:53.187064 24717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:40883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:40883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=41743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:24:53.189108 24717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:24:53.190902 24717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:24:53.220168 24723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:53.220244 24726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:53.220115 24724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:53.221292 24717 server_base.cc:1034] running on GCE node
I20260924 03:24:53.222592 24717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:24:53.224865 24717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:24:53.226192 24717 hybrid_clock.cc:648] HybridClock initialized: now 1790220293226115 us; error 67 us; skew 500 ppm
I20260924 03:24:53.226780 24717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:24:53 dist-test-slave-00qs krb5kdc[24163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220293, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:24:53.231572 24717 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:24:53.234016 24717 webserver.cc:460] Webserver started at http://127.10.12.190:41743/ using document root <none> and password file <none>
I20260924 03:24:53.235296 24717 fs_manager.cc:359] Metadata directory not provided
I20260924 03:24:53.235477 24717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:24:53.244254 24717 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 03:24:53.248739 24733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:53.250770 24717 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:24:53.251024 24717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "b5cdad64e69645e6b5135c0b65000576"
format_stamp: "Formatted at 2026-09-24 03:24:49 on dist-test-slave-00qs"
server_key: "5da1f558ac4c85fc7acb63cc4a61d6ef"
server_key_iv: "ba0eb310dde3e7264e2bb8b058137d77"
server_key_version: "encryptionkey@0"
I20260924 03:24:53.252468 24717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:24:53.278187 24717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:24:53.284808 24717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:24:53.285183 24717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:24:53.309762 24717 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:40883
I20260924 03:24:53.309821 24784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:40883 every 8 connection(s)
I20260924 03:24:53.311731 24717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:24:53.319305 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 24717
I20260924 03:24:53.320385 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:35751
--local_ip_for_outbound_sockets=127.10.12.129
--tserver_master_addrs=127.10.12.190:40883
--webserver_port=38227
--webserver_interface=127.10.12.129
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:24:53.321074 24785 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:24:53.325357 24785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576: Bootstrap starting.
I20260924 03:24:53.354225 24785 log.cc:824] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576: Log is configured to *not* fsync() on all Append() calls
I20260924 03:24:53.370146 24785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:24:53.370826 24785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576: Bootstrap complete.
I20260924 03:24:53.381367 24785 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5cdad64e69645e6b5135c0b65000576" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 40883 } }
I20260924 03:24:53.382471 24785 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5cdad64e69645e6b5135c0b65000576, State: Initialized, Role: FOLLOWER
I20260924 03:24:53.383394 24785 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5cdad64e69645e6b5135c0b65000576" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 40883 } }
I20260924 03:24:53.383695 24785 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:24:53.383831 24785 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:24:53.384063 24785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:24:53.388991 24785 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5cdad64e69645e6b5135c0b65000576" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 40883 } }
I20260924 03:24:53.389673 24785 leader_election.cc:302] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5cdad64e69645e6b5135c0b65000576; no voters:
I20260924 03:24:53.390419 24785 leader_election.cc:288] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 03:24:53.390596 24789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 03:24:53.391399 24789 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [term 2 LEADER]: Becoming Leader. State: Replica: b5cdad64e69645e6b5135c0b65000576, State: Running, Role: LEADER
I20260924 03:24:53.392139 24789 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5cdad64e69645e6b5135c0b65000576" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 40883 } }
I20260924 03:24:53.394104 24785 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:24:53.395128 24791 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b5cdad64e69645e6b5135c0b65000576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5cdad64e69645e6b5135c0b65000576" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 40883 } } }
I20260924 03:24:53.395159 24790 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5cdad64e69645e6b5135c0b65000576. Latest consensus state: current_term: 2 leader_uuid: "b5cdad64e69645e6b5135c0b65000576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5cdad64e69645e6b5135c0b65000576" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 40883 } } }
I20260924 03:24:53.395588 24791 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [sys.catalog]: This master's current role is: LEADER
I20260924 03:24:53.395593 24790 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [sys.catalog]: This master's current role is: LEADER
I20260924 03:24:53.407021 24794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:24:53.419744 24794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:24:53.421756 24794 catalog_manager.cc:1259] Loaded cluster ID: 7dc26a5347be49d39de2eb2e2bac1a7d
I20260924 03:24:53.421993 24794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:24:53.426198 24794 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 03:24:53.432394 24806 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 03:24:53.441259 24794 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 03:24:53.450433 24794 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:24:53.452255 24794 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576: Loaded TSK: 0
I20260924 03:24:53.453840 24794 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 03:24:53.776373 24787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:24:53.776793 24787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:24:53.776938 24787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:24:53.786247 24787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:24:53.786450 24787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:24:53.786618 24787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:24:53.800014 24787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:35751
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=38227
--tserver_master_addrs=127.10.12.190:40883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:24:53.801911 24787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:24:53.803725 24787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:24:53.824831 24812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:53.824831 24813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:53.825568 24815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:53.826628 24787 server_base.cc:1034] running on GCE node
I20260924 03:24:53.827476 24787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:24:53.829013 24787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:24:53.830283 24787 hybrid_clock.cc:648] HybridClock initialized: now 1790220293830223 us; error 60 us; skew 500 ppm
I20260924 03:24:53.830770 24787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:24:53 dist-test-slave-00qs krb5kdc[24163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220293, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:24:53.835187 24787 init.cc:377] Logged in from keytab as kudu/127.10.12.129@KRBTEST.COM (short username kudu)
I20260924 03:24:53.837584 24787 webserver.cc:460] Webserver started at http://127.10.12.129:38227/ using document root <none> and password file <none>
I20260924 03:24:53.838600 24787 fs_manager.cc:359] Metadata directory not provided
I20260924 03:24:53.838786 24787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:24:53.846879 24787 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:24:53.851012 24822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:53.852945 24787 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:24:53.853156 24787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "1f7843376981417a9629a0124f527e25"
format_stamp: "Formatted at 2026-09-24 03:24:50 on dist-test-slave-00qs"
server_key: "c99f92db7eac903e5eb01c9fa3ea69a8"
server_key_iv: "76a425d8b9cb072ecc2ec7919447475a"
server_key_version: "encryptionkey@0"
I20260924 03:24:53.854452 24787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:24:53.878858 24787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:24:53.884482 24787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:24:53.884804 24787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:24:53.886010 24787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:24:53.888304 24829 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260924 03:24:53.895084 24829 ts_tablet_manager.cc:1852] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 03:24:53.896450 24829 ts_tablet_manager.cc:1905] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:24:53.906029 24829 ts_tablet_manager.cc:1918] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:24:53.906308 24829 log.cc:1196] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/wal/wals/0b0accbfa7864d849bb008660c6ddd5f
I20260924 03:24:53.906920 24829 ts_tablet_manager.cc:1939] T 0b0accbfa7864d849bb008660c6ddd5f P 1f7843376981417a9629a0124f527e25: Deleting consensus metadata
I20260924 03:24:53.907426 24787 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 03:24:53.907575 24787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.020s user 0.000s sys 0.001s
I20260924 03:24:53.907781 24787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:24:53.907873 24787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:53.944895 24787 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:35751
I20260924 03:24:53.944924 24935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:35751 every 8 connection(s)
I20260924 03:24:53.946596 24787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:24:53.956171 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 24787
I20260924 03:24:53.957047 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:41635
--local_ip_for_outbound_sockets=127.10.12.130
--tserver_master_addrs=127.10.12.190:40883
--webserver_port=46655
--webserver_interface=127.10.12.130
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:24:53 dist-test-slave-00qs krb5kdc[24163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220293, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:24:53.982731 24938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:53.950667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:53957 (local address 127.10.12.190:40883)
0924 03:24:53.951489 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:53.951508 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:53.952431 (+ 923us) server_negotiation.cc:407] Connection header received
0924 03:24:53.955495 (+ 3064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:53.955511 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:53.956124 (+ 613us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:53.956557 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:53.959134 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:53.960900 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:53.963719 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:53.964726 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:53.964921 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:53.970175 (+ 5254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:53.970189 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:53.970217 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:53.970301 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:53.974588 (+ 4287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:53.976786 (+ 2198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:53.976797 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:53.976807 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:53.977106 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:53.977948 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:53.977954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:53.977957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:53.978848 (+ 891us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:53.979095 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:53.980692 (+ 1597us) server_negotiation.cc:299] Negotiation successful
0924 03:24:53.981232 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":336,"threads_started":1}
I20260924 03:24:53.982790 24939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:53.951117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.129:53957)
0924 03:24:53.952173 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:53.952236 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:53.954689 (+ 2453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:53.956834 (+ 2145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:53.956847 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:53.957757 (+ 910us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:24:53.958774 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:53.958843 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:53.961199 (+ 2356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:53.961205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:53.963465 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:53.963493 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:53.965048 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:53.965064 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:53.965383 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:53.966379 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:24:53.966417 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:24:53.969626 (+ 3209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:24:53.974993 (+ 5367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:53.975033 (+ 40us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:53.975061 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:53.976522 (+ 1461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:53.977279 (+ 757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:53.977285 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:53.977289 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:53.977767 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:53.979105 (+ 1338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:24:53.979117 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:24:53.979781 (+ 664us) client_negotiation.cc:769] Sending connection context
0924 03:24:53.980266 (+ 485us) client_negotiation.cc:240] Negotiation successful
0924 03:24:53.981075 (+ 809us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":384,"threads_started":1}
I20260924 03:24:53.988377 24936 heartbeater.cc:347] Connected to a master server at 127.10.12.190:40883
I20260924 03:24:53.988881 24936 heartbeater.cc:464] Registering TS with master...
I20260924 03:24:53.990504 24936 heartbeater.cc:511] Master 127.10.12.190:40883 requested a full tablet report, sending...
I20260924 03:24:53.994241 24750 ts_manager.cc:194] Registered new tserver with Master: 1f7843376981417a9629a0124f527e25 (127.10.12.129:35751)
I20260924 03:24:53.996855 24750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.129@KRBTEST.COM'} at 127.10.12.129:53957
W20260924 03:24:54.397256 24940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:24:54.397668 24940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:24:54.397820 24940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:24:54.407140 24940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:24:54.407326 24940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:24:54.407524 24940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:24:54.420481 24940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:41635
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=46655
--tserver_master_addrs=127.10.12.190:40883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:24:54.422267 24940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:24:54.424118 24940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:24:54.440837 24945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:54.440873 24948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:54.440804 24946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:54.442445 24940 server_base.cc:1034] running on GCE node
I20260924 03:24:54.443358 24940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:24:54.445024 24940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:24:54.446317 24940 hybrid_clock.cc:648] HybridClock initialized: now 1790220294446259 us; error 60 us; skew 500 ppm
I20260924 03:24:54.446753 24940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:24:54 dist-test-slave-00qs krb5kdc[24163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220294, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:24:54.451491 24940 init.cc:377] Logged in from keytab as kudu/127.10.12.130@KRBTEST.COM (short username kudu)
I20260924 03:24:54.453773 24940 webserver.cc:460] Webserver started at http://127.10.12.130:46655/ using document root <none> and password file <none>
I20260924 03:24:54.454828 24940 fs_manager.cc:359] Metadata directory not provided
I20260924 03:24:54.455018 24940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:24:54.463618 24940 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.002s
I20260924 03:24:54.468369 24955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:54.470723 24940 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260924 03:24:54.471004 24940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "fd88482e1b8d4a728220a423d383d7c0"
format_stamp: "Formatted at 2026-09-24 03:24:50 on dist-test-slave-00qs"
server_key: "59a20f0fee691e34fa98d0e768f4a1fa"
server_key_iv: "df6e565cef00b8fc2f9654c2727452b6"
server_key_version: "encryptionkey@0"
I20260924 03:24:54.472455 24940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:24:54.503077 24940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:24:54.509600 24940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:24:54.509958 24940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:24:54.511291 24940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:24:54.513137 24940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:24:54.513255 24940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:54.513387 24940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:24:54.513463 24940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:54.552320 24940 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:41635
I20260924 03:24:54.552387 25067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:41635 every 8 connection(s)
I20260924 03:24:54.554127 24940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:24:54.561297 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 24940
I20260924 03:24:54.562299 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:40173
--local_ip_for_outbound_sockets=127.10.12.131
--tserver_master_addrs=127.10.12.190:40883
--webserver_port=34165
--webserver_interface=127.10.12.131
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:24:54 dist-test-slave-00qs krb5kdc[24163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220294, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:24:54.582286 25070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:54.557356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:34373 (local address 127.10.12.190:40883)
0924 03:24:54.557955 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:54.557963 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:54.559079 (+ 1116us) server_negotiation.cc:407] Connection header received
0924 03:24:54.561562 (+ 2483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:54.561571 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:54.561710 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:54.561875 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:54.564246 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:54.565397 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:54.567658 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:54.568400 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:54.568593 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:54.574522 (+ 5929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:54.574529 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:54.574533 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:54.574599 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:54.577815 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:54.579021 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:54.579026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:54.579029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:54.579259 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:54.579853 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:54.579857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:54.579859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:54.580338 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:54.580477 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:54.581653 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0924 03:24:54.581929 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":293,"threads_started":1}
I20260924 03:24:54.584084 25071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:54.557984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.130:34373)
0924 03:24:54.558906 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:54.558952 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:54.561110 (+ 2158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:54.562250 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:54.562272 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:54.563128 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:24:54.563922 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:54.563946 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:54.565720 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:54.565726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:54.567380 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:54.567404 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:54.568863 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:54.568874 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:54.569218 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:54.570137 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:24:54.570170 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:24:54.574104 (+ 3934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:24:54.578018 (+ 3914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:54.578031 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:54.578056 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:54.578836 (+ 780us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:54.579382 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:54.579387 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:54.579390 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:54.579726 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:54.580474 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:24:54.580485 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:24:54.581111 (+ 626us) client_negotiation.cc:769] Sending connection context
0924 03:24:54.581493 (+ 382us) client_negotiation.cc:240] Negotiation successful
0924 03:24:54.582216 (+ 723us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":327,"threads_started":1}
I20260924 03:24:54.586354 25068 heartbeater.cc:347] Connected to a master server at 127.10.12.190:40883
I20260924 03:24:54.586769 25068 heartbeater.cc:464] Registering TS with master...
I20260924 03:24:54.587831 25068 heartbeater.cc:511] Master 127.10.12.190:40883 requested a full tablet report, sending...
I20260924 03:24:54.590061 24750 ts_manager.cc:194] Registered new tserver with Master: fd88482e1b8d4a728220a423d383d7c0 (127.10.12.130:41635)
I20260924 03:24:54.591068 24750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.130@KRBTEST.COM'} at 127.10.12.130:34373
W20260924 03:24:54.977777 25072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:24:54.978200 25072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:24:54.978351 25072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:24:54.986984 25072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:24:54.987174 25072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:24:54.987351 25072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:24:54.998915 25072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:40173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=34165
--tserver_master_addrs=127.10.12.190:40883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:24:55.000722 25072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:24:55.001425 24936 heartbeater.cc:503] Master 127.10.12.190:40883 was elected leader, sending a full tablet report...
I20260924 03:24:55.003072 25072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:24:55.020623 25077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:55.020597 25078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:55.020519 25080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:55.021868 25072 server_base.cc:1034] running on GCE node
I20260924 03:24:55.022728 25072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:24:55.024163 25072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:24:55.025406 25072 hybrid_clock.cc:648] HybridClock initialized: now 1790220295025345 us; error 58 us; skew 500 ppm
I20260924 03:24:55.025866 25072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:24:55 dist-test-slave-00qs krb5kdc[24163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220295, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:24:55.030146 25072 init.cc:377] Logged in from keytab as kudu/127.10.12.131@KRBTEST.COM (short username kudu)
I20260924 03:24:55.032202 25072 webserver.cc:460] Webserver started at http://127.10.12.131:34165/ using document root <none> and password file <none>
I20260924 03:24:55.033147 25072 fs_manager.cc:359] Metadata directory not provided
I20260924 03:24:55.033317 25072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:24:55.040421 25072 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:24:55.044286 25087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:55.046348 25072 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:24:55.046643 25072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "0559b9773b154613b98b7bd99a46f987"
format_stamp: "Formatted at 2026-09-24 03:24:51 on dist-test-slave-00qs"
server_key: "52c8c77eab9473ff0112f51bdadf04e8"
server_key_iv: "aade1740265a79dccd63b99042f3de9b"
server_key_version: "encryptionkey@0"
I20260924 03:24:55.048091 25072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:24:55.081482 25072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:24:55.087219 25072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:24:55.087558 25072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:24:55.088899 25072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:24:55.090694 25072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:24:55.090816 25072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:55.090950 25072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:24:55.091024 25072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:55.127310 25072 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:40173
I20260924 03:24:55.127324 25199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:40173 every 8 connection(s)
I20260924 03:24:55.129254 25072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:24:55.137979 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 25072
Sep 24 03:24:55 dist-test-slave-00qs krb5kdc[24163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220295, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:24:55.158363 25205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:55.139281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34732 (local address 127.10.12.190:40883)
0924 03:24:55.140392 (+ 1111us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:55.140399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:55.140420 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:24:55.140507 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:55.140513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:55.140605 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:55.140736 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:55.142607 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.143502 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.145312 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.145836 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.145968 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:55.149568 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:55.149579 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:55.149585 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:55.149651 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:55.154219 (+ 4568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:55.155545 (+ 1326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:55.155551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:55.155554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:55.155813 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:55.156289 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:55.156293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:55.156295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:55.156693 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:55.156994 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:55.157442 (+ 448us) server_negotiation.cc:299] Negotiation successful
0924 03:24:55.157742 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":892,"thread_start_us":654,"threads_started":1}
I20260924 03:24:55.158674 25202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:55.132529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:52777 (local address 127.10.12.190:40883)
0924 03:24:55.133162 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:55.133168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:55.134182 (+ 1014us) server_negotiation.cc:407] Connection header received
0924 03:24:55.136666 (+ 2484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:55.136673 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:55.136790 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:55.136908 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:55.139899 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.141190 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.143158 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.143651 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.143764 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:55.149655 (+ 5891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:55.149662 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:55.149666 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:55.149718 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:55.154261 (+ 4543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:55.155559 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:55.155565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:55.155568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:55.155769 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:55.156343 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:55.156347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:55.156349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:55.156685 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:55.156858 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:55.157856 (+ 998us) server_negotiation.cc:299] Negotiation successful
0924 03:24:55.158126 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":496,"spinlock_wait_cycles":2816,"thread_start_us":281,"threads_started":1}
I20260924 03:24:55.160127 25203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:55.133051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.131:52777)
0924 03:24:55.133942 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:55.134005 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:55.136193 (+ 2188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:55.137300 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:55.137314 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:55.138355 (+ 1041us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:24:55.139514 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:55.139562 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.141404 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.141409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:55.142977 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:55.142997 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.143943 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.143980 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:55.144290 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:55.145373 (+ 1083us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:24:55.145406 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:24:55.149057 (+ 3651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:24:55.154493 (+ 5436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:55.154506 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:55.154532 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:55.155441 (+ 909us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:55.155909 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:24:55.155914 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:24:55.155918 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:24:55.156223 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:24:55.156835 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:24:55.156845 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:24:55.157445 (+ 600us) client_negotiation.cc:769] Sending connection context
0924 03:24:55.157746 (+ 301us) client_negotiation.cc:240] Negotiation successful
0924 03:24:55.158499 (+ 753us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":327,"threads_started":1}
I20260924 03:24:55.161945 25200 heartbeater.cc:347] Connected to a master server at 127.10.12.190:40883
I20260924 03:24:55.162323 25200 heartbeater.cc:464] Registering TS with master...
I20260924 03:24:55.163278 25200 heartbeater.cc:511] Master 127.10.12.190:40883 requested a full tablet report, sending...
I20260924 03:24:55.165433 24750 ts_manager.cc:194] Registered new tserver with Master: 0559b9773b154613b98b7bd99a46f987 (127.10.12.131:40173)
I20260924 03:24:55.166438 24750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.131@KRBTEST.COM'} at 127.10.12.131:52777
I20260924 03:24:55.175179 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:24:55.184244 25205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:55.176568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34740 (local address 127.10.12.190:40883)
0924 03:24:55.176848 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:55.176855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:55.177362 (+ 507us) server_negotiation.cc:407] Connection header received
0924 03:24:55.177742 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:55.177750 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:55.177975 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:55.178153 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:24:55.178676 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.179503 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.181353 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.181890 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.182039 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:55.182690 (+ 651us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:24:55.183442 (+ 752us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:24:55.183614 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:55.183732 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 03:24:55.183857 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 03:24:55.185514 24750 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:34740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:24:55.194227 24750 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:24:55.228902 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:55.213184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35102 (local address 127.10.12.129:35751)
0924 03:24:55.213975 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:55.213989 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:55.218043 (+ 4054us) server_negotiation.cc:407] Connection header received
0924 03:24:55.218445 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:55.218455 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:55.218742 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:55.218952 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:24:55.220729 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.222309 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.225684 (+ 3375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.227336 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.227514 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:55.227858 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:55.228215 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 03:24:55.228453 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":2793,"threads_started":1}
I20260924 03:24:55.229429 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:55.212407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:35751 (local address 127.0.0.1:35102)
0924 03:24:55.217862 (+ 5455us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:55.217884 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:55.218193 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:55.219278 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:55.219282 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:55.219315 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:24:55.220473 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:55.220484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.222507 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.222512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:55.225427 (+ 2915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:55.225449 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.227615 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.227620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:55.227863 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:55.227912 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 03:24:55.228113 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 03:24:55.228679 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5304,"mutex_wait_us":85,"thread_start_us":5135,"threads_started":1}
I20260924 03:24:55.230417 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:55.211981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46616 (local address 127.10.12.131:40173)
0924 03:24:55.216755 (+ 4774us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:55.216768 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:55.217887 (+ 1119us) server_negotiation.cc:407] Connection header received
0924 03:24:55.218031 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:55.218041 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:55.218464 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:55.218731 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:24:55.220353 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.222609 (+ 2256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.226696 (+ 4087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.228437 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.228639 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:55.228925 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:55.229700 (+ 775us) server_negotiation.cc:299] Negotiation successful
0924 03:24:55.229969 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4526,"thread_start_us":4321,"threads_started":1}
I20260924 03:24:55.231546 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:55.211647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:40173 (local address 127.0.0.1:46616)
0924 03:24:55.217420 (+ 5773us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:55.217449 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:55.217709 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:55.219164 (+ 1455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:55.219174 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:55.219212 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:24:55.220078 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:55.220101 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.222870 (+ 2769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.222877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:55.226391 (+ 3514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:55.226412 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.229161 (+ 2749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.229169 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:55.229407 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:55.229453 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 03:24:55.230828 (+ 1375us) client_negotiation.cc:240] Negotiation successful
0924 03:24:55.231045 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5592,"mutex_wait_us":86,"spinlock_wait_cycles":1280,"thread_start_us":2896,"threads_started":1}
I20260924 03:24:55.233060 25213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:55.211066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:41635 (local address 127.0.0.1:55508)
0924 03:24:55.212711 (+ 1645us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:55.212789 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:55.213055 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:55.220646 (+ 7591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:55.220652 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:55.220688 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:24:55.221293 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:55.221314 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.223459 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.223466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:55.226391 (+ 2925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:55.226412 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.232043 (+ 5631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.232058 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:55.232328 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:55.232368 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 03:24:55.232486 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 03:24:55.232656 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1255,"mutex_wait_us":237,"thread_start_us":5925,"threads_started":1}
I20260924 03:24:55.233189 25212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:55.211294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55508 (local address 127.10.12.130:41635)
0924 03:24:55.219900 (+ 8606us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:55.219910 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:55.219943 (+ 33us) server_negotiation.cc:407] Connection header received
0924 03:24:55.220087 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:55.220097 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:55.220386 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:55.220656 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:24:55.221531 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.223162 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.228462 (+ 5300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.230147 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.232268 (+ 2121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:55.232555 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:55.232645 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 03:24:55.232864 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8434,"thread_start_us":8313,"threads_started":1}
I20260924 03:24:55.233654 24871 tablet_service.cc:1467] Processing CreateTablet for tablet b23366f10bf3415bb254a58cb67dd885 (DEFAULT_TABLE table=test-table [id=6cd5a53068284fb3950573259a8956e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:24:55.235457 24871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23366f10bf3415bb254a58cb67dd885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:24:55.236778 25135 tablet_service.cc:1467] Processing CreateTablet for tablet b23366f10bf3415bb254a58cb67dd885 (DEFAULT_TABLE table=test-table [id=6cd5a53068284fb3950573259a8956e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:24:55.237644 25003 tablet_service.cc:1467] Processing CreateTablet for tablet b23366f10bf3415bb254a58cb67dd885 (DEFAULT_TABLE table=test-table [id=6cd5a53068284fb3950573259a8956e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:24:55.238742 25135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23366f10bf3415bb254a58cb67dd885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:24:55.239293 25003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23366f10bf3415bb254a58cb67dd885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:24:55.246840 25217 tablet_bootstrap.cc:492] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25: Bootstrap starting.
I20260924 03:24:55.251539 25217 tablet_bootstrap.cc:654] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25: Neither blocks nor log segments found. Creating new log.
I20260924 03:24:55.252933 25217 log.cc:824] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25: Log is configured to *not* fsync() on all Append() calls
I20260924 03:24:55.256675 25217 tablet_bootstrap.cc:492] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25: No bootstrap required, opened a new log
I20260924 03:24:55.257045 25217 ts_tablet_manager.cc:1397] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260924 03:24:55.260340 25219 tablet_bootstrap.cc:492] T b23366f10bf3415bb254a58cb67dd885 P 0559b9773b154613b98b7bd99a46f987: Bootstrap starting.
I20260924 03:24:55.264501 25220 tablet_bootstrap.cc:492] T b23366f10bf3415bb254a58cb67dd885 P fd88482e1b8d4a728220a423d383d7c0: Bootstrap starting.
I20260924 03:24:55.265055 25219 tablet_bootstrap.cc:654] T b23366f10bf3415bb254a58cb67dd885 P 0559b9773b154613b98b7bd99a46f987: Neither blocks nor log segments found. Creating new log.
I20260924 03:24:55.266525 25219 log.cc:824] T b23366f10bf3415bb254a58cb67dd885 P 0559b9773b154613b98b7bd99a46f987: Log is configured to *not* fsync() on all Append() calls
I20260924 03:24:55.267738 25217 raft_consensus.cc:348] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd88482e1b8d4a728220a423d383d7c0" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 41635 } } peers { permanent_uuid: "0559b9773b154613b98b7bd99a46f987" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40173 } } peers { permanent_uuid: "1f7843376981417a9629a0124f527e25" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35751 } }
I20260924 03:24:55.268424 25217 raft_consensus.cc:374] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:24:55.268558 25217 raft_consensus.cc:729] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f7843376981417a9629a0124f527e25, State: Initialized, Role: FOLLOWER
I20260924 03:24:55.269433 25220 tablet_bootstrap.cc:654] T b23366f10bf3415bb254a58cb67dd885 P fd88482e1b8d4a728220a423d383d7c0: Neither blocks nor log segments found. Creating new log.
I20260924 03:24:55.269419 25217 consensus_queue.cc:260] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd88482e1b8d4a728220a423d383d7c0" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 41635 } } peers { permanent_uuid: "0559b9773b154613b98b7bd99a46f987" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40173 } } peers { permanent_uuid: "1f7843376981417a9629a0124f527e25" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35751 } }
I20260924 03:24:55.270582 25219 tablet_bootstrap.cc:492] T b23366f10bf3415bb254a58cb67dd885 P 0559b9773b154613b98b7bd99a46f987: No bootstrap required, opened a new log
I20260924 03:24:55.270964 25219 ts_tablet_manager.cc:1397] T b23366f10bf3415bb254a58cb67dd885 P 0559b9773b154613b98b7bd99a46f987: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260924 03:24:55.271636 25217 ts_tablet_manager.cc:1428] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260924 03:24:55.273044 25220 log.cc:824] T b23366f10bf3415bb254a58cb67dd885 P fd88482e1b8d4a728220a423d383d7c0: Log is configured to *not* fsync() on all Append() calls
I20260924 03:24:55.277155 25220 tablet_bootstrap.cc:492] T b23366f10bf3415bb254a58cb67dd885 P fd88482e1b8d4a728220a423d383d7c0: No bootstrap required, opened a new log
I20260924 03:24:55.277554 25220 ts_tablet_manager.cc:1397] T b23366f10bf3415bb254a58cb67dd885 P fd88482e1b8d4a728220a423d383d7c0: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.000s
I20260924 03:24:55.281118 25219 raft_consensus.cc:348] T b23366f10bf3415bb254a58cb67dd885 P 0559b9773b154613b98b7bd99a46f987 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd88482e1b8d4a728220a423d383d7c0" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 41635 } } peers { permanent_uuid: "0559b9773b154613b98b7bd99a46f987" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40173 } } peers { permanent_uuid: "1f7843376981417a9629a0124f527e25" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35751 } }
I20260924 03:24:55.281591 25219 raft_consensus.cc:374] T b23366f10bf3415bb254a58cb67dd885 P 0559b9773b154613b98b7bd99a46f987 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:24:55.281693 25219 raft_consensus.cc:729] T b23366f10bf3415bb254a58cb67dd885 P 0559b9773b154613b98b7bd99a46f987 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0559b9773b154613b98b7bd99a46f987, State: Initialized, Role: FOLLOWER
I20260924 03:24:55.282368 25219 consensus_queue.cc:260] T b23366f10bf3415bb254a58cb67dd885 P 0559b9773b154613b98b7bd99a46f987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd88482e1b8d4a728220a423d383d7c0" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 41635 } } peers { permanent_uuid: "0559b9773b154613b98b7bd99a46f987" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40173 } } peers { permanent_uuid: "1f7843376981417a9629a0124f527e25" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35751 } }
I20260924 03:24:55.283658 25200 heartbeater.cc:503] Master 127.10.12.190:40883 was elected leader, sending a full tablet report...
I20260924 03:24:55.284289 25219 ts_tablet_manager.cc:1428] T b23366f10bf3415bb254a58cb67dd885 P 0559b9773b154613b98b7bd99a46f987: Time spent starting tablet: real 0.013s user 0.015s sys 0.001s
I20260924 03:24:55.287181 25220 raft_consensus.cc:348] T b23366f10bf3415bb254a58cb67dd885 P fd88482e1b8d4a728220a423d383d7c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd88482e1b8d4a728220a423d383d7c0" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 41635 } } peers { permanent_uuid: "0559b9773b154613b98b7bd99a46f987" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40173 } } peers { permanent_uuid: "1f7843376981417a9629a0124f527e25" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35751 } }
I20260924 03:24:55.287782 25220 raft_consensus.cc:374] T b23366f10bf3415bb254a58cb67dd885 P fd88482e1b8d4a728220a423d383d7c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:24:55.287899 25220 raft_consensus.cc:729] T b23366f10bf3415bb254a58cb67dd885 P fd88482e1b8d4a728220a423d383d7c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd88482e1b8d4a728220a423d383d7c0, State: Initialized, Role: FOLLOWER
I20260924 03:24:55.288566 25220 consensus_queue.cc:260] T b23366f10bf3415bb254a58cb67dd885 P fd88482e1b8d4a728220a423d383d7c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd88482e1b8d4a728220a423d383d7c0" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 41635 } } peers { permanent_uuid: "0559b9773b154613b98b7bd99a46f987" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40173 } } peers { permanent_uuid: "1f7843376981417a9629a0124f527e25" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35751 } }
I20260924 03:24:55.289696 25068 heartbeater.cc:503] Master 127.10.12.190:40883 was elected leader, sending a full tablet report...
I20260924 03:24:55.290306 25220 ts_tablet_manager.cc:1428] T b23366f10bf3415bb254a58cb67dd885 P fd88482e1b8d4a728220a423d383d7c0: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20260924 03:24:55.306603 25069 tablet.cc:2369] T b23366f10bf3415bb254a58cb67dd885 P fd88482e1b8d4a728220a423d383d7c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:24:55.324779 25222 raft_consensus.cc:482] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:24:55.325105 25222 raft_consensus.cc:504] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd88482e1b8d4a728220a423d383d7c0" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 41635 } } peers { permanent_uuid: "0559b9773b154613b98b7bd99a46f987" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40173 } } peers { permanent_uuid: "1f7843376981417a9629a0124f527e25" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35751 } }
I20260924 03:24:55.326977 25222 leader_election.cc:288] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd88482e1b8d4a728220a423d383d7c0 (127.10.12.130:41635), 0559b9773b154613b98b7bd99a46f987 (127.10.12.131:40173)
I20260924 03:24:55.337198 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:55.327562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:43625 (local address 127.10.12.131:40173)
0924 03:24:55.327759 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:55.327765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:55.328477 (+ 712us) server_negotiation.cc:407] Connection header received
0924 03:24:55.328683 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:55.328689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:55.328771 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:55.328908 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:24:55.329798 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.331058 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.334387 (+ 3329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.335683 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.335854 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:55.335939 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:55.336747 (+ 808us) server_negotiation.cc:299] Negotiation successful
0924 03:24:55.336954 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 03:24:55.337172 25212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:55.327472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:41573 (local address 127.10.12.130:41635)
0924 03:24:55.327698 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:55.327720 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:55.328540 (+ 820us) server_negotiation.cc:407] Connection header received
0924 03:24:55.328683 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:55.328689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:55.328771 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:55.328883 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:24:55.329820 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.331023 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.334403 (+ 3380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.335667 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.335854 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:55.335939 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:55.336642 (+ 703us) server_negotiation.cc:299] Negotiation successful
0924 03:24:55.336876 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 03:24:55.337373 25227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:55.327323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:40173 (local address 127.10.12.129:43625)
0924 03:24:55.328327 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:55.328349 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:55.328543 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:55.328970 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:55.328976 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:55.329027 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:24:55.329575 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:55.329595 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.331195 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.331201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:55.334023 (+ 2822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:55.334047 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.336047 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.336053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:55.336278 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:55.336342 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 03:24:55.336494 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 03:24:55.336626 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":175,"spinlock_wait_cycles":20096,"thread_start_us":667,"threads_started":1}
I20260924 03:24:55.337823 25226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:55.327430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:41635 (local address 127.10.12.129:41573)
0924 03:24:55.328426 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:55.328440 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:24:55.328548 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:24:55.328970 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:24:55.328976 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:24:55.329030 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:24:55.329634 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:55.329643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.331195 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.331201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:55.334027 (+ 2826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:24:55.334047 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.336028 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.336037 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:24:55.336283 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:55.336339 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 03:24:55.336457 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 03:24:55.336626 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"mutex_wait_us":285,"spinlock_wait_cycles":14464,"thread_start_us":470,"threads_started":1}
I20260924 03:24:55.339041 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23366f10bf3415bb254a58cb67dd885" candidate_uuid: "1f7843376981417a9629a0124f527e25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd88482e1b8d4a728220a423d383d7c0" is_pre_election: true
I20260924 03:24:55.338973 25155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23366f10bf3415bb254a58cb67dd885" candidate_uuid: "1f7843376981417a9629a0124f527e25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0559b9773b154613b98b7bd99a46f987" is_pre_election: true
I20260924 03:24:55.339582 25023 raft_consensus.cc:2454] T b23366f10bf3415bb254a58cb67dd885 P fd88482e1b8d4a728220a423d383d7c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f7843376981417a9629a0124f527e25 in term 0.
I20260924 03:24:55.339743 25155 raft_consensus.cc:2454] T b23366f10bf3415bb254a58cb67dd885 P 0559b9773b154613b98b7bd99a46f987 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f7843376981417a9629a0124f527e25 in term 0.
I20260924 03:24:55.340552 24826 leader_election.cc:302] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f7843376981417a9629a0124f527e25, fd88482e1b8d4a728220a423d383d7c0; no voters:
I20260924 03:24:55.341012 25222 raft_consensus.cc:2781] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:24:55.341149 25222 raft_consensus.cc:482] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:24:55.341252 25222 raft_consensus.cc:3037] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:24:55.343709 25222 raft_consensus.cc:504] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd88482e1b8d4a728220a423d383d7c0" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 41635 } } peers { permanent_uuid: "0559b9773b154613b98b7bd99a46f987" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40173 } } peers { permanent_uuid: "1f7843376981417a9629a0124f527e25" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35751 } }
I20260924 03:24:55.344439 25222 leader_election.cc:288] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25 [CANDIDATE]: Term 1 election: Requested vote from peers fd88482e1b8d4a728220a423d383d7c0 (127.10.12.130:41635), 0559b9773b154613b98b7bd99a46f987 (127.10.12.131:40173)
I20260924 03:24:55.345036 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23366f10bf3415bb254a58cb67dd885" candidate_uuid: "1f7843376981417a9629a0124f527e25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd88482e1b8d4a728220a423d383d7c0"
I20260924 03:24:55.345042 25155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23366f10bf3415bb254a58cb67dd885" candidate_uuid: "1f7843376981417a9629a0124f527e25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0559b9773b154613b98b7bd99a46f987"
I20260924 03:24:55.345245 25023 raft_consensus.cc:3037] T b23366f10bf3415bb254a58cb67dd885 P fd88482e1b8d4a728220a423d383d7c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:24:55.345239 25155 raft_consensus.cc:3037] T b23366f10bf3415bb254a58cb67dd885 P 0559b9773b154613b98b7bd99a46f987 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:24:55.347919 25155 raft_consensus.cc:2454] T b23366f10bf3415bb254a58cb67dd885 P 0559b9773b154613b98b7bd99a46f987 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f7843376981417a9629a0124f527e25 in term 1.
I20260924 03:24:55.348304 25023 raft_consensus.cc:2454] T b23366f10bf3415bb254a58cb67dd885 P fd88482e1b8d4a728220a423d383d7c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f7843376981417a9629a0124f527e25 in term 1.
I20260924 03:24:55.348397 24823 leader_election.cc:302] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0559b9773b154613b98b7bd99a46f987, 1f7843376981417a9629a0124f527e25; no voters:
I20260924 03:24:55.348788 25222 raft_consensus.cc:2781] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:24:55.349344 25222 raft_consensus.cc:686] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25 [term 1 LEADER]: Becoming Leader. State: Replica: 1f7843376981417a9629a0124f527e25, State: Running, Role: LEADER
I20260924 03:24:55.349893 25222 consensus_queue.cc:237] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd88482e1b8d4a728220a423d383d7c0" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 41635 } } peers { permanent_uuid: "0559b9773b154613b98b7bd99a46f987" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40173 } } peers { permanent_uuid: "1f7843376981417a9629a0124f527e25" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35751 } }
I20260924 03:24:55.355453 24749 catalog_manager.cc:5495] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f7843376981417a9629a0124f527e25 (127.10.12.129). New cstate: current_term: 1 leader_uuid: "1f7843376981417a9629a0124f527e25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd88482e1b8d4a728220a423d383d7c0" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 41635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0559b9773b154613b98b7bd99a46f987" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f7843376981417a9629a0124f527e25" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35751 } health_report { overall_health: HEALTHY } } }
W20260924 03:24:55.381172 25201 tablet.cc:2369] T b23366f10bf3415bb254a58cb67dd885 P 0559b9773b154613b98b7bd99a46f987: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:24:55.390048 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:55.384296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35104 (local address 127.10.12.129:35751)
0924 03:24:55.384528 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:55.384535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:55.384607 (+ 72us) server_negotiation.cc:407] Connection header received
0924 03:24:55.384869 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:55.384873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:55.384956 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:55.385095 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:24:55.385492 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.386242 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.387698 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.388165 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.388279 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:55.388883 (+ 604us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:24:55.389431 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:24:55.389579 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:55.389682 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 03:24:55.389773 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 03:24:55.399564 25155 raft_consensus.cc:1261] T b23366f10bf3415bb254a58cb67dd885 P 0559b9773b154613b98b7bd99a46f987 [term 1 FOLLOWER]: Refusing update from remote peer 1f7843376981417a9629a0124f527e25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:24:55.399564 25023 raft_consensus.cc:1261] T b23366f10bf3415bb254a58cb67dd885 P fd88482e1b8d4a728220a423d383d7c0 [term 1 FOLLOWER]: Refusing update from remote peer 1f7843376981417a9629a0124f527e25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:24:55.400642 25228 consensus_queue.cc:1035] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd88482e1b8d4a728220a423d383d7c0" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 41635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:24:55.401052 25222 consensus_queue.cc:1035] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0559b9773b154613b98b7bd99a46f987" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:24:55.429394 24750 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:34740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:24:55.429692 24750 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:34740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:24:55.433686 24750 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576: Sending DeleteTablet for 3 replicas of tablet b23366f10bf3415bb254a58cb67dd885
I20260924 03:24:55.435070 24871 tablet_service.cc:1514] Processing DeleteTablet for tablet b23366f10bf3415bb254a58cb67dd885 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:24:55 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:35102
I20260924 03:24:55.435045 25135 tablet_service.cc:1514] Processing DeleteTablet for tablet b23366f10bf3415bb254a58cb67dd885 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:24:55 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:46616
I20260924 03:24:55.435905 25003 tablet_service.cc:1514] Processing DeleteTablet for tablet b23366f10bf3415bb254a58cb67dd885 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:24:55 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:55508
I20260924 03:24:55.436101 25241 tablet_replica.cc:330] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25: stopping tablet replica
I20260924 03:24:55.436589 25241 raft_consensus.cc:2229] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:24:55.437788 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 24787
I20260924 03:24:55.437347 25241 raft_consensus.cc:2258] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:24:55.439219 25240 tablet_replica.cc:330] T b23366f10bf3415bb254a58cb67dd885 P 0559b9773b154613b98b7bd99a46f987: stopping tablet replica
I20260924 03:24:55.440060 25242 tablet_replica.cc:330] T b23366f10bf3415bb254a58cb67dd885 P fd88482e1b8d4a728220a423d383d7c0: stopping tablet replica
I20260924 03:24:55.440573 25242 raft_consensus.cc:2229] T b23366f10bf3415bb254a58cb67dd885 P fd88482e1b8d4a728220a423d383d7c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:24:55.441700 25241 ts_tablet_manager.cc:1905] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:24:55.441131 25242 raft_consensus.cc:2258] T b23366f10bf3415bb254a58cb67dd885 P fd88482e1b8d4a728220a423d383d7c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:24:55.444248 25242 ts_tablet_manager.cc:1905] T b23366f10bf3415bb254a58cb67dd885 P fd88482e1b8d4a728220a423d383d7c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:24:55.448469 25240 raft_consensus.cc:2229] T b23366f10bf3415bb254a58cb67dd885 P 0559b9773b154613b98b7bd99a46f987 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:24:55.449144 25240 raft_consensus.cc:2258] T b23366f10bf3415bb254a58cb67dd885 P 0559b9773b154613b98b7bd99a46f987 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:24:55.451558 25240 ts_tablet_manager.cc:1905] T b23366f10bf3415bb254a58cb67dd885 P 0559b9773b154613b98b7bd99a46f987: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:24:55.451717 25242 ts_tablet_manager.cc:1918] T b23366f10bf3415bb254a58cb67dd885 P fd88482e1b8d4a728220a423d383d7c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:24:55.452010 25242 log.cc:1196] T b23366f10bf3415bb254a58cb67dd885 P fd88482e1b8d4a728220a423d383d7c0: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/b23366f10bf3415bb254a58cb67dd885
I20260924 03:24:55.452632 25242 ts_tablet_manager.cc:1939] T b23366f10bf3415bb254a58cb67dd885 P fd88482e1b8d4a728220a423d383d7c0: Deleting consensus metadata
I20260924 03:24:55.452867 25241 ts_tablet_manager.cc:1918] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:24:55.453059 25241 log.cc:1196] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/wal/wals/b23366f10bf3415bb254a58cb67dd885
I20260924 03:24:55.453675 25241 ts_tablet_manager.cc:1939] T b23366f10bf3415bb254a58cb67dd885 P 1f7843376981417a9629a0124f527e25: Deleting consensus metadata
I20260924 03:24:55.455039 24737 catalog_manager.cc:4841] TS fd88482e1b8d4a728220a423d383d7c0 (127.10.12.130:41635): tablet b23366f10bf3415bb254a58cb67dd885 (table test-table [id=6cd5a53068284fb3950573259a8956e2]) successfully deleted
I20260924 03:24:55.456554 24736 catalog_manager.cc:4841] TS 1f7843376981417a9629a0124f527e25 (127.10.12.129:35751): tablet b23366f10bf3415bb254a58cb67dd885 (table test-table [id=6cd5a53068284fb3950573259a8956e2]) successfully deleted
I20260924 03:24:55.458014 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:55.438657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35106 (local address 127.10.12.129:35751)
0924 03:24:55.438829 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:55.438835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:55.438971 (+ 136us) server_negotiation.cc:407] Connection header received
0924 03:24:55.439196 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:55.439202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:55.439293 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:55.439390 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:55.441269 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.442624 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.444589 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.445219 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.445350 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:55.449407 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:55.449426 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:55.449447 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:55.449499 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:55.453818 (+ 4319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:55.454853 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:55.454864 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:55.454868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:55.455136 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:55.455683 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:55.455687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:55.455690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:55.456262 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:55.456580 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:55.457250 (+ 670us) server_negotiation.cc:299] Negotiation successful
0924 03:24:55.457542 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 03:24:55.458758 24931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:24:55.458978 25240 ts_tablet_manager.cc:1918] T b23366f10bf3415bb254a58cb67dd885 P 0559b9773b154613b98b7bd99a46f987: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:24:55.459173 25240 log.cc:1196] T b23366f10bf3415bb254a58cb67dd885 P 0559b9773b154613b98b7bd99a46f987: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/b23366f10bf3415bb254a58cb67dd885
I20260924 03:24:55.459765 25240 ts_tablet_manager.cc:1939] T b23366f10bf3415bb254a58cb67dd885 P 0559b9773b154613b98b7bd99a46f987: Deleting consensus metadata
I20260924 03:24:55.461910 24734 catalog_manager.cc:4841] TS 0559b9773b154613b98b7bd99a46f987 (127.10.12.131:40173): tablet b23366f10bf3415bb254a58cb67dd885 (table test-table [id=6cd5a53068284fb3950573259a8956e2]) successfully deleted
I20260924 03:24:55.562788 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 24787
I20260924 03:24:55.579871 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 24940
I20260924 03:24:55.595047 25212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:55.580801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55514 (local address 127.10.12.130:41635)
0924 03:24:55.581003 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:55.581009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:55.581031 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:24:55.581095 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:55.581101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:55.581165 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:55.581249 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:55.582649 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.583402 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.584832 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.585232 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.585344 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:55.588033 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:55.588053 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:55.588072 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:55.588125 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:55.591842 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:55.592737 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:55.592747 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:55.592750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:55.592908 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:55.593302 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:55.593307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:55.593309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:55.593782 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:55.594091 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:55.594459 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 03:24:55.594697 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 03:24:55.595834 25063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:24:55.686018 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 24940
I20260924 03:24:55.700593 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 25072
I20260924 03:24:55.715308 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:55.701478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46624 (local address 127.10.12.131:40173)
0924 03:24:55.701657 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 03:24:55.701661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:24:55.701681 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:24:55.701859 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:24:55.701867 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:24:55.701964 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:24:55.702063 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:24:55.703586 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.704525 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.705919 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:24:55.706326 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:24:55.706441 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:24:55.709365 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:24:55.709385 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:24:55.709402 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:24:55.709443 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:24:55.712397 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:55.713184 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:55.713192 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:55.713194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:55.713332 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:24:55.713700 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:24:55.713703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:24:55.713705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:24:55.714087 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:24:55.714316 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 03:24:55.714772 (+ 456us) server_negotiation.cc:299] Negotiation successful
0924 03:24:55.714969 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:24:55.716034 25195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:24:55.807345 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 25072
I20260924 03:24:55.820974 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 24717
I20260924 03:24:55.822183 24780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:24:55.942068 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 24717
I20260924 03:24:55.961187 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:40883
--webserver_interface=127.10.12.190
--webserver_port=41743
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:40883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:24:56.322350 25247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:24:56.322777 25247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:24:56.322916 25247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:24:56.331360 25247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:24:56.331528 25247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:24:56.331594 25247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:24:56.331687 25247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:24:56.331826 25247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:24:56.344161 25247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:40883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:40883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=41743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:24:56.346100 25247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:24:56.347831 25247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:24:56.362515 25252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:56.362432 25255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:56.362620 25253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:56.363618 25247 server_base.cc:1034] running on GCE node
I20260924 03:24:56.365003 25247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:24:56.367399 25247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:24:56.368712 25247 hybrid_clock.cc:648] HybridClock initialized: now 1790220296368624 us; error 93 us; skew 500 ppm
I20260924 03:24:56.369213 25247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:24:56 dist-test-slave-00qs krb5kdc[24163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220296, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:24:56.373294 25247 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:24:56.375427 25247 webserver.cc:460] Webserver started at http://127.10.12.190:41743/ using document root <none> and password file <none>
I20260924 03:24:56.376477 25247 fs_manager.cc:359] Metadata directory not provided
I20260924 03:24:56.376641 25247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:24:56.384410 25247 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 03:24:56.390717 25262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:56.392871 25247 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 03:24:56.393157 25247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "b5cdad64e69645e6b5135c0b65000576"
format_stamp: "Formatted at 2026-09-24 03:24:49 on dist-test-slave-00qs"
server_key: "5da1f558ac4c85fc7acb63cc4a61d6ef"
server_key_iv: "ba0eb310dde3e7264e2bb8b058137d77"
server_key_version: "encryptionkey@0"
I20260924 03:24:56.394493 25247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:24:56.433210 25247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:24:56.438943 25247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:24:56.439267 25247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:24:56.461966 25247 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:40883
I20260924 03:24:56.461995 25313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:40883 every 8 connection(s)
I20260924 03:24:56.464160 25247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:24:56.467402 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 25247
I20260924 03:24:56.468626 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:35751
--local_ip_for_outbound_sockets=127.10.12.129
--tserver_master_addrs=127.10.12.190:40883
--webserver_port=38227
--webserver_interface=127.10.12.129
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:24:56.476257 25314 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:24:56.481694 25314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576: Bootstrap starting.
I20260924 03:24:56.508044 25314 log.cc:824] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576: Log is configured to *not* fsync() on all Append() calls
I20260924 03:24:56.528481 25314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:24:56.529217 25314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576: Bootstrap complete.
I20260924 03:24:56.539855 25314 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5cdad64e69645e6b5135c0b65000576" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 40883 } }
I20260924 03:24:56.541038 25314 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5cdad64e69645e6b5135c0b65000576, State: Initialized, Role: FOLLOWER
I20260924 03:24:56.541810 25314 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5cdad64e69645e6b5135c0b65000576" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 40883 } }
I20260924 03:24:56.542088 25314 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:24:56.542207 25314 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:24:56.542405 25314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [term 2 FOLLOWER]: Advancing to term 3
I20260924 03:24:56.547181 25314 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5cdad64e69645e6b5135c0b65000576" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 40883 } }
I20260924 03:24:56.547889 25314 leader_election.cc:302] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5cdad64e69645e6b5135c0b65000576; no voters:
I20260924 03:24:56.548870 25314 leader_election.cc:288] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 03:24:56.549271 25318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 03:24:56.553966 25318 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [term 3 LEADER]: Becoming Leader. State: Replica: b5cdad64e69645e6b5135c0b65000576, State: Running, Role: LEADER
I20260924 03:24:56.554822 25318 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5cdad64e69645e6b5135c0b65000576" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 40883 } }
I20260924 03:24:56.555322 25314 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:24:56.558183 25320 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5cdad64e69645e6b5135c0b65000576. Latest consensus state: current_term: 3 leader_uuid: "b5cdad64e69645e6b5135c0b65000576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5cdad64e69645e6b5135c0b65000576" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 40883 } } }
I20260924 03:24:56.558732 25320 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [sys.catalog]: This master's current role is: LEADER
I20260924 03:24:56.558573 25319 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b5cdad64e69645e6b5135c0b65000576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5cdad64e69645e6b5135c0b65000576" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 40883 } } }
I20260924 03:24:56.558928 25319 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5cdad64e69645e6b5135c0b65000576 [sys.catalog]: This master's current role is: LEADER
I20260924 03:24:56.569923 25327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:24:56.582931 25327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:24:56.585815 25327 catalog_manager.cc:1259] Loaded cluster ID: 7dc26a5347be49d39de2eb2e2bac1a7d
I20260924 03:24:56.586117 25327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 03:24:56.590432 25327 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f899d0e99ad google::LogMessage::Fail() at ??:0
@ 0x7f899d0ec88a google::LogMessage::SendToLog() at ??:0
@ 0x7f899d0e94de google::LogMessage::Flush() at ??:0
@ 0x7f899d0eaa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f89b2b1c512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f89b2b19976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f89b2ba1610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f89b2ba1439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f899fd35345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f899fd3891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f899fd38799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f899fd13eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f89a2fed6db start_thread at ??:0
@ 0x7f899bf8361f clone at ??:0
W20260924 03:24:56.974092 25316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:24:56.974499 25316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:24:56.974637 25316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:24:56.983016 25316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:24:56.983191 25316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:24:56.983352 25316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:24:56.994988 25316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:35751
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=38227
--tserver_master_addrs=127.10.12.190:40883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:24:56.996852 25316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:24:56.998535 25316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:24:57.015620 25355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:57.015676 25358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:57.015569 25356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:57.017067 25316 server_base.cc:1034] running on GCE node
I20260924 03:24:57.017936 25316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:24:57.019300 25316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:24:57.020546 25316 hybrid_clock.cc:648] HybridClock initialized: now 1790220297020490 us; error 43 us; skew 500 ppm
I20260924 03:24:57.020970 25316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:24:57 dist-test-slave-00qs krb5kdc[24163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220297, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:24:57.025345 25316 init.cc:377] Logged in from keytab as kudu/127.10.12.129@KRBTEST.COM (short username kudu)
I20260924 03:24:57.027501 25316 webserver.cc:460] Webserver started at http://127.10.12.129:38227/ using document root <none> and password file <none>
I20260924 03:24:57.028506 25316 fs_manager.cc:359] Metadata directory not provided
I20260924 03:24:57.028644 25316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:24:57.036242 25316 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.003s
I20260924 03:24:57.040174 25365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:57.042230 25316 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:24:57.042472 25316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "1f7843376981417a9629a0124f527e25"
format_stamp: "Formatted at 2026-09-24 03:24:50 on dist-test-slave-00qs"
server_key: "c99f92db7eac903e5eb01c9fa3ea69a8"
server_key_iv: "76a425d8b9cb072ecc2ec7919447475a"
server_key_version: "encryptionkey@0"
I20260924 03:24:57.043784 25316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:24:57.076028 25316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:24:57.082093 25316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:24:57.082481 25316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:24:57.083807 25316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:24:57.085573 25316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:24:57.085707 25316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:57.085824 25316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:24:57.085897 25316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:57.122674 25316 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:35751
I20260924 03:24:57.122747 25477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:35751 every 8 connection(s)
I20260924 03:24:57.124797 25316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:24:57.126981 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 25316
I20260924 03:24:57.127802 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:41635
--local_ip_for_outbound_sockets=127.10.12.130
--tserver_master_addrs=127.10.12.190:40883
--webserver_port=46655
--webserver_interface=127.10.12.130
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:24:57.140807 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:57.129084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.129:47819)
0924 03:24:57.138803 (+ 9719us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:57.139256 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9120,"thread_start_us":8777,"threads_started":1}
W20260924 03:24:57.141877 25478 heartbeater.cc:644] Failed to heartbeat to 127.10.12.190:40883 (0 consecutive failures): Network error: Failed to ping master at 127.10.12.190:40883: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
I20260924 03:24:57.143122 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:57.142645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.129:47461)
0924 03:24:57.142847 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:57.142958 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 03:24:57.144403 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:57.143835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.129:33091)
0924 03:24:57.144099 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:57.144225 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 03:24:57.144790 25478 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:24:57.558292 25481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:24:57.558729 25481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:24:57.558911 25481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:24:57.567719 25481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:24:57.567982 25481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:24:57.568192 25481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:24:57.580246 25481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:41635
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=46655
--tserver_master_addrs=127.10.12.190:40883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:24:57.582299 25481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:24:57.584211 25481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:24:57.602325 25487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:57.602315 25486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:57.602563 25489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:57.603169 25481 server_base.cc:1034] running on GCE node
I20260924 03:24:57.604053 25481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:24:57.605329 25481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:24:57.606564 25481 hybrid_clock.cc:648] HybridClock initialized: now 1790220297606509 us; error 48 us; skew 500 ppm
I20260924 03:24:57.606943 25481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:24:57 dist-test-slave-00qs krb5kdc[24163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220297, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:24:57.616828 25481 init.cc:377] Logged in from keytab as kudu/127.10.12.130@KRBTEST.COM (short username kudu)
I20260924 03:24:57.619274 25481 webserver.cc:460] Webserver started at http://127.10.12.130:46655/ using document root <none> and password file <none>
I20260924 03:24:57.620699 25481 fs_manager.cc:359] Metadata directory not provided
I20260924 03:24:57.620891 25481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:24:57.629204 25481 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 03:24:57.633687 25496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:57.635831 25481 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260924 03:24:57.636153 25481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "fd88482e1b8d4a728220a423d383d7c0"
format_stamp: "Formatted at 2026-09-24 03:24:50 on dist-test-slave-00qs"
server_key: "59a20f0fee691e34fa98d0e768f4a1fa"
server_key_iv: "df6e565cef00b8fc2f9654c2727452b6"
server_key_version: "encryptionkey@0"
I20260924 03:24:57.637598 25481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:24:57.666805 25481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:24:57.672801 25481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:24:57.673162 25481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:24:57.674495 25481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:24:57.676307 25481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:24:57.676445 25481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:57.676584 25481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:24:57.676671 25481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:57.717303 25481 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:41635
I20260924 03:24:57.717458 25608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:41635 every 8 connection(s)
I20260924 03:24:57.719365 25481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:24:57.725471 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 25481
I20260924 03:24:57.726415 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:40173
--local_ip_for_outbound_sockets=127.10.12.131
--tserver_master_addrs=127.10.12.190:40883
--webserver_port=34165
--webserver_interface=127.10.12.131
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:24:57.726454 25611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:57.723518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.130:47595)
0924 03:24:57.724744 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:57.725080 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":802,"thread_start_us":508,"threads_started":1}
W20260924 03:24:57.727667 25609 heartbeater.cc:644] Failed to heartbeat to 127.10.12.190:40883 (0 consecutive failures): Network error: Failed to ping master at 127.10.12.190:40883: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
I20260924 03:24:57.729099 25611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:57.728565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.130:45563)
0924 03:24:57.728808 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:57.728920 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260924 03:24:57.738669 25611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:57.737979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.130:34091)
0924 03:24:57.738275 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:57.738422 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260924 03:24:57.739148 25609 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 03:24:58.146781 25613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:58.145692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.129:34405)
0924 03:24:58.146354 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:58.146482 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":341,"threads_started":1}
W20260924 03:24:58.157830 25612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:24:58.158290 25612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:24:58.158429 25612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:24:58.167012 25612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:24:58.167214 25612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:24:58.167403 25612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:24:58.178944 25612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:40173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=34165
--tserver_master_addrs=127.10.12.190:40883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:24:58.180903 25612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:24:58.182704 25612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:24:58.200498 25618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:58.200528 25619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:24:58.200766 25621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:24:58.202558 25612 server_base.cc:1034] running on GCE node
I20260924 03:24:58.203548 25612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:24:58.204969 25612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:24:58.206213 25612 hybrid_clock.cc:648] HybridClock initialized: now 1790220298206145 us; error 54 us; skew 500 ppm
I20260924 03:24:58.206606 25612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:24:58 dist-test-slave-00qs krb5kdc[24163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220298, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:24:58.210659 25612 init.cc:377] Logged in from keytab as kudu/127.10.12.131@KRBTEST.COM (short username kudu)
I20260924 03:24:58.212797 25612 webserver.cc:460] Webserver started at http://127.10.12.131:34165/ using document root <none> and password file <none>
I20260924 03:24:58.213743 25612 fs_manager.cc:359] Metadata directory not provided
I20260924 03:24:58.213892 25612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:24:58.221403 25612 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 03:24:58.225319 25628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:58.227367 25612 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 03:24:58.227696 25612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "0559b9773b154613b98b7bd99a46f987"
format_stamp: "Formatted at 2026-09-24 03:24:51 on dist-test-slave-00qs"
server_key: "52c8c77eab9473ff0112f51bdadf04e8"
server_key_iv: "aade1740265a79dccd63b99042f3de9b"
server_key_version: "encryptionkey@0"
I20260924 03:24:58.229076 25612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:24:58.255726 25612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:24:58.261816 25612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:24:58.262216 25612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:24:58.263558 25612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:24:58.265540 25612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:24:58.265691 25612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:58.265821 25612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:24:58.265897 25612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:24:58.304934 25612 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:40173
I20260924 03:24:58.305006 25740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:40173 every 8 connection(s)
I20260924 03:24:58.306830 25612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:24:58.313046 25743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:58.310525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.131:44797)
0924 03:24:58.311521 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:58.311816 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":409,"threads_started":1}
I20260924 03:24:58.313781 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 25612
W20260924 03:24:58.313967 25741 heartbeater.cc:644] Failed to heartbeat to 127.10.12.190:40883 (0 consecutive failures): Network error: Failed to ping master at 127.10.12.190:40883: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
I20260924 03:24:58.315122 25743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:58.314642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.131:42125)
0924 03:24:58.314855 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:58.314952 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260924 03:24:58.316501 25743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:58.315990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.131:55997)
0924 03:24:58.316272 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:58.316359 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
W20260924 03:24:58.316833 25741 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 03:24:58.741125 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:58.740016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.130:53495)
0924 03:24:58.740630 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:58.740819 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":269,"threads_started":1}
I20260924 03:24:59.149391 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:59.148131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.129:40875)
0924 03:24:59.148982 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:59.149153 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":557,"threads_started":1}
I20260924 03:24:59.318516 25747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:59.317487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.131:42275)
0924 03:24:59.318041 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:59.318215 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":266,"threads_started":1}
I20260924 03:24:59.743445 25748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:24:59.742472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.130:50043)
0924 03:24:59.743082 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0924 03:24:59.743218 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":290,"threads_started":1}
I20260924 03:25:00.151684 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:00.150600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.129:54619)
0924 03:25:00.151305 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:00.151423 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":309,"threads_started":1}
I20260924 03:25:00.320834 25750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:00.319902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.131:50161)
0924 03:25:00.320520 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:00.320621 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":298,"threads_started":1}
I20260924 03:25:00.745441 25751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:00.744528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.130:38527)
0924 03:25:00.745103 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:00.745210 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":317,"threads_started":1}
I20260924 03:25:01.153996 25752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:01.152937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.129:57693)
0924 03:25:01.153513 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:01.153708 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":301,"threads_started":1}
I20260924 03:25:01.322867 25753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:01.321911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.131:35511)
0924 03:25:01.322471 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:01.322577 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":283,"threads_started":1}
I20260924 03:25:01.747790 25754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:01.746675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.130:57559)
0924 03:25:01.747413 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:01.747547 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":383,"threads_started":1}
I20260924 03:25:02.156463 25755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:02.155372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.129:33879)
0924 03:25:02.156098 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:02.156220 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":366,"threads_started":1}
I20260924 03:25:02.325289 25756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:02.324250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.131:35621)
0924 03:25:02.324934 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:02.325048 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":329,"threads_started":1}
I20260924 03:25:02.752926 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:02.749379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.130:38511)
0924 03:25:02.752524 (+ 3145us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:02.752664 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2997,"thread_start_us":2779,"threads_started":1}
I20260924 03:25:03.158996 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:03.157886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.129:37711)
0924 03:25:03.158534 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:03.158771 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":342,"threads_started":1}
I20260924 03:25:03.327935 25759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:03.326851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.131:59373)
0924 03:25:03.327520 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:03.327701 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":339,"threads_started":1}
I20260924 03:25:03.755616 25760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:03.754432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.130:49777)
0924 03:25:03.755153 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:03.755290 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":423,"threads_started":1}
I20260924 03:25:04.161752 25761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:04.160540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.129:44829)
0924 03:25:04.161258 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:04.161390 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":321,"threads_started":1}
I20260924 03:25:04.330338 25762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:04.329271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.131:51561)
0924 03:25:04.329983 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:04.330105 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":428,"threads_started":1}
I20260924 03:25:04.758315 25763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:04.757087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.130:52871)
0924 03:25:04.757889 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:04.758028 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":371,"threads_started":1}
I20260924 03:25:05.164485 25764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:05.163200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.129:46739)
0924 03:25:05.164066 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:05.164214 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":455,"threads_started":1}
I20260924 03:25:05.332696 25765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:05.331564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.131:38703)
0924 03:25:05.332283 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:05.332405 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":319,"threads_started":1}
I20260924 03:25:05.761047 25766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:05.759899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.130:59293)
0924 03:25:05.760648 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:05.760802 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":311,"threads_started":1}
I20260924 03:25:06.167245 25767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:06.166073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.129:53877)
0924 03:25:06.166796 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:06.166967 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":359,"threads_started":1}
I20260924 03:25:06.335099 25768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:06.334075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.131:42591)
0924 03:25:06.334742 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:06.334874 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":365,"threads_started":1}
I20260924 03:25:06.763397 25769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:06.762345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.130:42789)
0924 03:25:06.763051 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:06.763171 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":378,"threads_started":1}
I20260924 03:25:07.169865 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:07.168841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.129:41891)
0924 03:25:07.169508 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:07.169632 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":370,"threads_started":1}
I20260924 03:25:07.337405 25771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:07.336334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.131:55769)
0924 03:25:07.337029 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:07.337167 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":323,"threads_started":1}
I20260924 03:25:07.765812 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:07.764807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.130:50283)
0924 03:25:07.765451 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:07.765557 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":370,"threads_started":1}
I20260924 03:25:08.172334 25773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:08.171245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.129:35133)
0924 03:25:08.171873 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:08.172069 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":326,"threads_started":1}
I20260924 03:25:08.339783 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:08.338827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.131:46069)
0924 03:25:08.339439 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:08.339564 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":335,"threads_started":1}
I20260924 03:25:08.768446 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:08.767176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.130:46355)
0924 03:25:08.767882 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:08.768117 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":309,"threads_started":1}
I20260924 03:25:09.174749 25776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:09.173751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.129:60875)
0924 03:25:09.174360 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:09.174478 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":309,"threads_started":1}
I20260924 03:25:09.342336 25777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:09.341241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.131:41009)
0924 03:25:09.341912 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:09.342077 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":317,"threads_started":1}
I20260924 03:25:09.770895 25778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:09.769912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.130:56135)
0924 03:25:09.770573 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:09.770684 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":397,"threads_started":1}
I20260924 03:25:10.177430 25779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:10.176233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.129:54355)
0924 03:25:10.176960 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:10.177145 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":352,"threads_started":1}
I20260924 03:25:10.344691 25780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:10.343726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.131:44917)
0924 03:25:10.344371 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:10.344477 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":358,"threads_started":1}
I20260924 03:25:10.773509 25781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:10.772375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.130:60181)
0924 03:25:10.773097 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:10.773258 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":394,"threads_started":1}
I20260924 03:25:11.179904 25782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:11.178810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.129:57763)
0924 03:25:11.179459 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:11.179595 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":314,"threads_started":1}
I20260924 03:25:11.346899 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:11.345882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.131:34885)
0924 03:25:11.346527 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:11.346644 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":293,"threads_started":1}
I20260924 03:25:11.775722 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:11.774791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.130:37427)
0924 03:25:11.775384 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:11.775484 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":310,"threads_started":1}
I20260924 03:25:12.183099 25785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:12.181952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.129:53441)
0924 03:25:12.182660 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:12.182856 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":376,"threads_started":1}
I20260924 03:25:12.349164 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:12.348097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.131:51681)
0924 03:25:12.348782 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:12.348895 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":360,"threads_started":1}
I20260924 03:25:12.778220 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:12.777218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.130:38851)
0924 03:25:12.777828 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:12.777967 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":312,"threads_started":1}
I20260924 03:25:13.185392 25788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:13.184405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.129:56261)
0924 03:25:13.185066 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:13.185181 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":360,"threads_started":1}
I20260924 03:25:13.316202 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 25316
I20260924 03:25:13.340162 25789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:13.317756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46576 (local address 127.10.12.129:35751)
0924 03:25:13.318570 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:13.318593 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:13.318657 (+ 64us) server_negotiation.cc:407] Connection header received
0924 03:25:13.320934 (+ 2277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:13.320950 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:13.321604 (+ 654us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:13.322056 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:25:13.324038 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:13.325560 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:13.327063 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:13.328293 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:13.328521 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:13.331763 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:25:13.331774 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:25:13.331797 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:25:13.331864 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:25:13.335719 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:13.336799 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:13.336809 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:13.336818 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:13.337027 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:13.337447 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:13.337451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:13.337454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:13.338211 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:25:13.338580 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:13.339192 (+ 612us) server_negotiation.cc:299] Negotiation successful
0924 03:25:13.339756 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":401,"threads_started":1}
I20260924 03:25:13.341930 25473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:25:13.351630 25791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:13.350692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:40883 (local address 127.10.12.131:55405)
0924 03:25:13.351277 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:13.351408 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":295,"threads_started":1}
I20260924 03:25:13.436299 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 25316
I20260924 03:25:13.455931 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 25481
I20260924 03:25:13.478791 25792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:13.457316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.10.12.130:41635)
0924 03:25:13.458031 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:13.458048 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:13.458100 (+ 52us) server_negotiation.cc:407] Connection header received
0924 03:25:13.460152 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:13.460167 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:13.460785 (+ 618us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:13.461247 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:25:13.463034 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:13.464625 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:13.466119 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:13.467003 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:13.467229 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:13.470647 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:25:13.470659 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:25:13.470685 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:25:13.470746 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:25:13.474327 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:13.475272 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:13.475282 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:13.475291 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:13.475517 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:13.475945 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:13.476037 (+ 92us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:13.476039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:13.476774 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:25:13.477037 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:13.477862 (+ 825us) server_negotiation.cc:299] Negotiation successful
0924 03:25:13.478441 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":344,"threads_started":1}
I20260924 03:25:13.480661 25604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:25:13.574506 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 25481
I20260924 03:25:13.593004 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 25612
I20260924 03:25:13.615370 25794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:13.594316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55388 (local address 127.10.12.131:40173)
0924 03:25:13.595066 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:13.595084 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:13.595137 (+ 53us) server_negotiation.cc:407] Connection header received
0924 03:25:13.597183 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:13.597198 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:13.597856 (+ 658us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:13.598277 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:25:13.600223 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:13.601727 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:13.603272 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:13.604344 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:13.604546 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:13.607662 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:25:13.607673 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:25:13.607696 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:25:13.607750 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:25:13.611261 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:13.612319 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:13.612330 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:13.612341 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:13.612605 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:13.613048 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:13.613053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:13.613055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:13.613793 (+ 738us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:25:13.614078 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:13.614506 (+ 428us) server_negotiation.cc:299] Negotiation successful
0924 03:25:13.615008 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":371,"threads_started":1}
I20260924 03:25:13.617254 25736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:25:13.712198 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 25612
2026-09-24T03:25:13Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25906 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:25:13 dist-test-slave-00qs krb5kdc[25798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:25:13 dist-test-slave-00qs krb5kdc[25798](info): set up 2 sockets
Sep 24 03:25:13 dist-test-slave-00qs krb5kdc[25798](info): commencing operation
krb5kdc: starting...
W20260924 03:25:14.781209 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:25:14 dist-test-slave-00qs krb5kdc[25798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:25:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:25:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:25:15.081000 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:39019
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:39019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:25:15.496687 25811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:25:15.497253 25811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:25:15.497444 25811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:25:15.506413 25811 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 03:25:15.506573 25811 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 03:25:15.506620 25811 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 03:25:15.506711 25811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:25:15.506793 25811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:25:15.506897 25811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:25:15.506968 25811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:25:15.507047 25811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:25:15.519053 25811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:39019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:39019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:25:15.521008 25811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:25:15.522912 25811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:25:15.538379 25816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:25:15.538419 25819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:25:15.538341 25817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:25:15.539474 25811 server_base.cc:1034] running on GCE node
I20260924 03:25:15.540755 25811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:25:15.542902 25811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:25:15.544238 25811 hybrid_clock.cc:648] HybridClock initialized: now 1790220315544175 us; error 79 us; skew 500 ppm
I20260924 03:25:15.544852 25811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:25:15 dist-test-slave-00qs krb5kdc[25798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220315, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:25:15.549088 25811 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:25:15.551110 25811 webserver.cc:460] Webserver started at http://127.10.12.190:32933/ using document root <none> and password file <none>
I20260924 03:25:15.552084 25811 fs_manager.cc:359] Metadata directory not provided
I20260924 03:25:15.552210 25811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:25:15.552582 25811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:25:15.556471 25811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "3f1b56edf4c14316aae772d84db31fef"
format_stamp: "Formatted at 2026-09-24 03:25:15 on dist-test-slave-00qs"
server_key: "3e9c374c4bb2685da4f70d477b0154a1"
server_key_iv: "d0da3792bb788181fdb9e952f2f4fd4f"
server_key_version: "encryptionkey@0"
I20260924 03:25:15.557462 25811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "3f1b56edf4c14316aae772d84db31fef"
format_stamp: "Formatted at 2026-09-24 03:25:15 on dist-test-slave-00qs"
server_key: "3e9c374c4bb2685da4f70d477b0154a1"
server_key_iv: "d0da3792bb788181fdb9e952f2f4fd4f"
server_key_version: "encryptionkey@0"
I20260924 03:25:15.564679 25811 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260924 03:25:15.569762 25826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:25:15.571784 25811 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260924 03:25:15.572121 25811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "3f1b56edf4c14316aae772d84db31fef"
format_stamp: "Formatted at 2026-09-24 03:25:15 on dist-test-slave-00qs"
server_key: "3e9c374c4bb2685da4f70d477b0154a1"
server_key_iv: "d0da3792bb788181fdb9e952f2f4fd4f"
server_key_version: "encryptionkey@0"
I20260924 03:25:15.572431 25811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:25:15.597426 25811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:25:15.603335 25811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:25:15.603682 25811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:25:15.625803 25811 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:39019
I20260924 03:25:15.625784 25877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:39019 every 8 connection(s)
I20260924 03:25:15.627693 25811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:25:15.633203 25878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:25:15.634269 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 25811
I20260924 03:25:15.634538 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:25:15.635500 10290 external_mini_cluster.cc:1428] Setting key 14b61d66619842778edd276d512b7e8b
I20260924 03:25:15.649159 25878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef: Bootstrap starting.
I20260924 03:25:15.653735 25878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef: Neither blocks nor log segments found. Creating new log.
Sep 24 03:25:15 dist-test-slave-00qs krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:25:15.655207 25878 log.cc:824] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef: Log is configured to *not* fsync() on all Append() calls
I20260924 03:25:15.659909 25878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef: No bootstrap required, opened a new log
I20260924 03:25:15.667291 25881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:15.639293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39334 (local address 127.10.12.190:39019)
0924 03:25:15.640442 (+ 1149us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:15.640462 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:15.640524 (+ 62us) server_negotiation.cc:407] Connection header received
0924 03:25:15.642636 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:15.642671 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:15.643405 (+ 734us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:15.644024 (+ 619us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:25:15.646443 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:15.648280 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:15.650157 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:15.651079 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:15.651346 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:15.656602 (+ 5256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:25:15.656616 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:25:15.656647 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:25:15.656707 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:25:15.661149 (+ 4442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:15.662161 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:15.662177 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:15.662190 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:15.662481 (+ 291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:15.662965 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:15.662971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:15.662974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:15.663927 (+ 953us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:25:15.664343 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:15.664835 (+ 492us) server_negotiation.cc:299] Negotiation successful
0924 03:25:15.665450 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":826,"thread_start_us":551,"threads_started":1}
I20260924 03:25:15.670910 25878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b56edf4c14316aae772d84db31fef" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39019 } }
I20260924 03:25:15.671373 25878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:25:15.671480 25878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1b56edf4c14316aae772d84db31fef, State: Initialized, Role: FOLLOWER
I20260924 03:25:15.672158 25878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b56edf4c14316aae772d84db31fef" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39019 } }
I20260924 03:25:15.672426 25878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:25:15.672555 25878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:25:15.672727 25878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:25:15.675426 25878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b56edf4c14316aae772d84db31fef" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39019 } }
I20260924 03:25:15.676133 25878 leader_election.cc:302] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f1b56edf4c14316aae772d84db31fef; no voters:
I20260924 03:25:15.676853 25878 leader_election.cc:288] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:25:15.677237 25883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:25:15.678056 25883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [term 1 LEADER]: Becoming Leader. State: Replica: 3f1b56edf4c14316aae772d84db31fef, State: Running, Role: LEADER
I20260924 03:25:15.678823 25883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b56edf4c14316aae772d84db31fef" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39019 } }
I20260924 03:25:15.679827 25878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:25:15.682144 25885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f1b56edf4c14316aae772d84db31fef. Latest consensus state: current_term: 1 leader_uuid: "3f1b56edf4c14316aae772d84db31fef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b56edf4c14316aae772d84db31fef" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39019 } } }
I20260924 03:25:15.682646 25885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [sys.catalog]: This master's current role is: LEADER
I20260924 03:25:15.685233 25884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f1b56edf4c14316aae772d84db31fef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b56edf4c14316aae772d84db31fef" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39019 } } }
I20260924 03:25:15.685672 25884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [sys.catalog]: This master's current role is: LEADER
I20260924 03:25:15.685858 25890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:25:15.694455 25890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:25:15.709649 25890 catalog_manager.cc:1347] Generated new cluster ID: 682e1b919ea04febbbdd19dc3c5e5caa
I20260924 03:25:15.709848 25890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:25:15.724073 25890 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:25:15.725951 25890 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:25:15.736137 25890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef: Generated new TSK 0
I20260924 03:25:15.737474 25890 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
I20260924 03:25:15.817817 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:39019
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:25:16.192704 25904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:25:16.193033 25904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:25:16.193157 25904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:25:16.201081 25904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:25:16.201216 25904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:25:16.201366 25904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:25:16.212107 25904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:39019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:25:16.213544 25904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:25:16.214941 25904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:25:16.229331 25909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:25:16.229250 25912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:25:16.229308 25910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:25:16.236073 25904 server_base.cc:1034] running on GCE node
I20260924 03:25:16.236855 25904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:25:16.237932 25904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:25:16.239194 25904 hybrid_clock.cc:648] HybridClock initialized: now 1790220316239132 us; error 56 us; skew 500 ppm
I20260924 03:25:16.239554 25904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:25:16 dist-test-slave-00qs krb5kdc[25798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220316, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:25:16.242946 25904 init.cc:377] Logged in from keytab as kudu/127.10.12.129@KRBTEST.COM (short username kudu)
I20260924 03:25:16.244861 25904 webserver.cc:460] Webserver started at http://127.10.12.129:44953/ using document root <none> and password file <none>
I20260924 03:25:16.245613 25904 fs_manager.cc:359] Metadata directory not provided
I20260924 03:25:16.245719 25904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:25:16.246080 25904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:25:16.249210 25904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "4aa96ce3582e4c9abc4290645cd71562"
format_stamp: "Formatted at 2026-09-24 03:25:16 on dist-test-slave-00qs"
server_key: "342f08ff6e5e065deeefb9bf431aad93"
server_key_iv: "34a07bfef0433181c695ab07c3687584"
server_key_version: "encryptionkey@0"
I20260924 03:25:16.250016 25904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "4aa96ce3582e4c9abc4290645cd71562"
format_stamp: "Formatted at 2026-09-24 03:25:16 on dist-test-slave-00qs"
server_key: "342f08ff6e5e065deeefb9bf431aad93"
server_key_iv: "34a07bfef0433181c695ab07c3687584"
server_key_version: "encryptionkey@0"
I20260924 03:25:16.255676 25904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 03:25:16.259603 25919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:25:16.261201 25904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:25:16.261370 25904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "4aa96ce3582e4c9abc4290645cd71562"
format_stamp: "Formatted at 2026-09-24 03:25:16 on dist-test-slave-00qs"
server_key: "342f08ff6e5e065deeefb9bf431aad93"
server_key_iv: "34a07bfef0433181c695ab07c3687584"
server_key_version: "encryptionkey@0"
I20260924 03:25:16.261602 25904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:25:16.291895 25904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:25:16.296586 25904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:25:16.296883 25904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:25:16.297926 25904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:25:16.299388 25904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:25:16.299471 25904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:25:16.299576 25904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:25:16.299633 25904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:25:16.333715 25904 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:42927
I20260924 03:25:16.333784 26031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:42927 every 8 connection(s)
I20260924 03:25:16.335052 25904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:25:16.336500 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 25904
I20260924 03:25:16.336721 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:25:16.337559 10290 external_mini_cluster.cc:1428] Setting key 1e0522d544742c77c4c59395693087b9
Sep 24 03:25:16 dist-test-slave-00qs krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220316, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:25:16.358367 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:16.338169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:42963 (local address 127.10.12.190:39019)
0924 03:25:16.338643 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:16.338649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:16.340129 (+ 1480us) server_negotiation.cc:407] Connection header received
0924 03:25:16.342172 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:16.342178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:16.342283 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:16.342399 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:25:16.344170 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:16.345182 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:16.347482 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:16.347918 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:16.348086 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:16.352092 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:25:16.352099 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:25:16.352102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:25:16.352145 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:25:16.354868 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:16.355758 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:16.355763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:16.355766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:16.355903 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:16.356393 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:16.356397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:16.356399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:16.356692 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:25:16.356832 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:16.357684 (+ 852us) server_negotiation.cc:299] Negotiation successful
0924 03:25:16.357940 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":229,"threads_started":1}
WARNING: no policy specified for kudu/127.10.12.130@KRBTEST.COM; defaulting to no policy
I20260924 03:25:16.359506 26035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:16.339003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.129:42963)
0924 03:25:16.339928 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:16.340014 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0924 03:25:16.341841 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:25:16.342637 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:25:16.342647 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:25:16.343265 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:25:16.343925 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:16.343946 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:16.345372 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:16.345377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:16.347159 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:16.347177 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:16.348193 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:16.348201 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:16.348456 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:16.349148 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:25:16.349175 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:25:16.351755 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:25:16.355068 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:25:16.355078 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:25:16.355095 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:25:16.355612 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:25:16.356043 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:25:16.356049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:25:16.356053 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:25:16.356263 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:25:16.356847 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:25:16.356857 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:25:16.357223 (+ 366us) client_negotiation.cc:769] Sending connection context
0924 03:25:16.357545 (+ 322us) client_negotiation.cc:240] Negotiation successful
0924 03:25:16.358174 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":430,"threads_started":1}
I20260924 03:25:16.361627 26032 heartbeater.cc:347] Connected to a master server at 127.10.12.190:39019
I20260924 03:25:16.361953 26032 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.130@KRBTEST.COM" created.
I20260924 03:25:16.362794 26032 heartbeater.cc:511] Master 127.10.12.190:39019 requested a full tablet report, sending...
I20260924 03:25:16.365312 25843 ts_manager.cc:194] Registered new tserver with Master: 4aa96ce3582e4c9abc4290645cd71562 (127.10.12.129:42927)
I20260924 03:25:16.367295 25843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.129@KRBTEST.COM'} at 127.10.12.129:42963
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
I20260924 03:25:16.393018 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:39019
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:25:16.729604 26038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:25:16.729911 26038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:25:16.730013 26038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:25:16.737664 26038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:25:16.737803 26038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:25:16.737952 26038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:25:16.748231 26038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:39019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:25:16.749651 26038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:25:16.751024 26038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:25:16.764513 26043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:25:16.765688 26044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:25:16.766585 26046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:25:16.766865 26038 server_base.cc:1034] running on GCE node
I20260924 03:25:16.767490 26038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:25:16.768523 26038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:25:16.769742 26038 hybrid_clock.cc:648] HybridClock initialized: now 1790220316769664 us; error 65 us; skew 500 ppm
I20260924 03:25:16.770090 26038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:25:16 dist-test-slave-00qs krb5kdc[25798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220316, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:25:16.773559 26038 init.cc:377] Logged in from keytab as kudu/127.10.12.130@KRBTEST.COM (short username kudu)
I20260924 03:25:16.775067 26038 webserver.cc:460] Webserver started at http://127.10.12.130:44653/ using document root <none> and password file <none>
I20260924 03:25:16.775835 26038 fs_manager.cc:359] Metadata directory not provided
I20260924 03:25:16.775940 26038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:25:16.776237 26038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:25:16.779235 26038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "a5d08a1e9e764220811433108eb7dcbe"
format_stamp: "Formatted at 2026-09-24 03:25:16 on dist-test-slave-00qs"
server_key: "329f2709cd00c5ba8317072c59970e7d"
server_key_iv: "fa5778828f770eebe6b1105185f46267"
server_key_version: "encryptionkey@0"
I20260924 03:25:16.780102 26038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "a5d08a1e9e764220811433108eb7dcbe"
format_stamp: "Formatted at 2026-09-24 03:25:16 on dist-test-slave-00qs"
server_key: "329f2709cd00c5ba8317072c59970e7d"
server_key_iv: "fa5778828f770eebe6b1105185f46267"
server_key_version: "encryptionkey@0"
I20260924 03:25:16.787796 26038 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260924 03:25:16.792850 26053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:25:16.794957 26038 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:25:16.795202 26038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "a5d08a1e9e764220811433108eb7dcbe"
format_stamp: "Formatted at 2026-09-24 03:25:16 on dist-test-slave-00qs"
server_key: "329f2709cd00c5ba8317072c59970e7d"
server_key_iv: "fa5778828f770eebe6b1105185f46267"
server_key_version: "encryptionkey@0"
I20260924 03:25:16.795467 26038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:25:16.813467 26038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:25:16.819201 26038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:25:16.819532 26038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:25:16.820818 26038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:25:16.822424 26038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:25:16.822515 26038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:25:16.822661 26038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:25:16.822715 26038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:25:16.861029 26038 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:44323
I20260924 03:25:16.861027 26165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:44323 every 8 connection(s)
I20260924 03:25:16.862929 26038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:25:16.867683 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 26038
I20260924 03:25:16.867904 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:25:16.868544 10290 external_mini_cluster.cc:1428] Setting key 18b50d23e72aef90a93d2d0673bd2457
Sep 24 03:25:16 dist-test-slave-00qs krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220316, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:25:16.890390 26168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:16.866555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:54597 (local address 127.10.12.190:39019)
0924 03:25:16.867130 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:16.867136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:16.868473 (+ 1337us) server_negotiation.cc:407] Connection header received
0924 03:25:16.870895 (+ 2422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:16.870899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:16.870989 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:16.871121 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:25:16.873242 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:16.874340 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:16.876454 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:16.877220 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:16.877386 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:16.882761 (+ 5375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:25:16.882767 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:25:16.882771 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:25:16.882827 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:25:16.886030 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:16.887231 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:16.887237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:16.887240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:16.887432 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:16.888049 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:16.888054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:16.888057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:16.888446 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:25:16.888612 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:16.889560 (+ 948us) server_negotiation.cc:299] Negotiation successful
0924 03:25:16.889913 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":303,"threads_started":1}
I20260924 03:25:16.891853 26169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:16.867180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.130:54597)
0924 03:25:16.868235 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:16.868299 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 03:25:16.870545 (+ 2246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:25:16.871409 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:25:16.871420 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:25:16.872218 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:25:16.873012 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:16.873035 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:16.874544 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:16.874549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:16.876254 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:16.876276 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:16.877548 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:16.877554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:16.877963 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:16.878926 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:25:16.878963 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:25:16.882339 (+ 3376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:25:16.886304 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:25:16.886315 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:25:16.886338 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:25:16.887042 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:25:16.887564 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:25:16.887570 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:25:16.887574 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:25:16.887878 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:25:16.888597 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:25:16.888609 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:25:16.889110 (+ 501us) client_negotiation.cc:769] Sending connection context
0924 03:25:16.889420 (+ 310us) client_negotiation.cc:240] Negotiation successful
0924 03:25:16.890219 (+ 799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":397,"threads_started":1}
I20260924 03:25:16.894208 26166 heartbeater.cc:347] Connected to a master server at 127.10.12.190:39019
I20260924 03:25:16.894608 26166 heartbeater.cc:464] Registering TS with master...
I20260924 03:25:16.895603 26166 heartbeater.cc:511] Master 127.10.12.190:39019 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.12.131@KRBTEST.COM; defaulting to no policy
I20260924 03:25:16.897775 25843 ts_manager.cc:194] Registered new tserver with Master: a5d08a1e9e764220811433108eb7dcbe (127.10.12.130:44323)
I20260924 03:25:16.898898 25843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.130@KRBTEST.COM'} at 127.10.12.130:54597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
I20260924 03:25:16.933847 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:39019
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:25:17.357133 26172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:25:17.357550 26172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:25:17.357679 26172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:25:17.366106 26172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:25:17.366286 26172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:25:17.366433 26172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:25:17.370990 26032 heartbeater.cc:503] Master 127.10.12.190:39019 was elected leader, sending a full tablet report...
I20260924 03:25:17.378809 26172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:39019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:25:17.380831 26172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:25:17.382678 26172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:25:17.401536 26177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:25:17.401536 26178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:25:17.401556 26180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:25:17.403669 26172 server_base.cc:1034] running on GCE node
I20260924 03:25:17.404662 26172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:25:17.406083 26172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:25:17.407339 26172 hybrid_clock.cc:648] HybridClock initialized: now 1790220317407276 us; error 66 us; skew 500 ppm
I20260924 03:25:17.407747 26172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:25:17 dist-test-slave-00qs krb5kdc[25798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220317, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:25:17.412490 26172 init.cc:377] Logged in from keytab as kudu/127.10.12.131@KRBTEST.COM (short username kudu)
I20260924 03:25:17.414964 26172 webserver.cc:460] Webserver started at http://127.10.12.131:35227/ using document root <none> and password file <none>
I20260924 03:25:17.416038 26172 fs_manager.cc:359] Metadata directory not provided
I20260924 03:25:17.416188 26172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:25:17.416584 26172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:25:17.420652 26172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "cd9dfeb5e9404919b0c4d22a61a6518a"
format_stamp: "Formatted at 2026-09-24 03:25:17 on dist-test-slave-00qs"
server_key: "bba68c1301e128a321885e4ecdcae5fd"
server_key_iv: "a2797c99994ccd33becb02efd70ac660"
server_key_version: "encryptionkey@0"
I20260924 03:25:17.421684 26172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "cd9dfeb5e9404919b0c4d22a61a6518a"
format_stamp: "Formatted at 2026-09-24 03:25:17 on dist-test-slave-00qs"
server_key: "bba68c1301e128a321885e4ecdcae5fd"
server_key_iv: "a2797c99994ccd33becb02efd70ac660"
server_key_version: "encryptionkey@0"
I20260924 03:25:17.428925 26172 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260924 03:25:17.434736 26187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:25:17.437081 26172 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260924 03:25:17.437336 26172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "cd9dfeb5e9404919b0c4d22a61a6518a"
format_stamp: "Formatted at 2026-09-24 03:25:17 on dist-test-slave-00qs"
server_key: "bba68c1301e128a321885e4ecdcae5fd"
server_key_iv: "a2797c99994ccd33becb02efd70ac660"
server_key_version: "encryptionkey@0"
I20260924 03:25:17.437628 26172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:25:17.477104 26172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:25:17.483407 26172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:25:17.483753 26172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:25:17.485131 26172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:25:17.486923 26172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:25:17.487010 26172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:25:17.487160 26172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:25:17.487219 26172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:25:17.523823 26172 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:40007
I20260924 03:25:17.523869 26299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:40007 every 8 connection(s)
I20260924 03:25:17.525558 26172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:25:17.527974 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 26172
I20260924 03:25:17.528226 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:25:17.528759 10290 external_mini_cluster.cc:1428] Setting key 918ca6392bcb02890ba27464e7e0cfd7
Sep 24 03:25:17 dist-test-slave-00qs krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220317, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:25:17.551219 26302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:17.529132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:39171 (local address 127.10.12.190:39019)
0924 03:25:17.529721 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:17.529729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:17.530952 (+ 1223us) server_negotiation.cc:407] Connection header received
0924 03:25:17.533553 (+ 2601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:17.533568 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:17.533679 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:17.533785 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:25:17.535909 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:17.537075 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:17.538971 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:17.539442 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:17.539579 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:17.544305 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:25:17.544311 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:25:17.544315 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:25:17.544360 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:25:17.547294 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:17.548279 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:17.548286 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:17.548289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:17.548439 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:17.548994 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:17.548998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:17.549001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:17.549351 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:25:17.549515 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:17.550459 (+ 944us) server_negotiation.cc:299] Negotiation successful
0924 03:25:17.550762 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":261,"threads_started":1}
I20260924 03:25:17.552578 26303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:17.529830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.131:39171)
0924 03:25:17.530766 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:17.530834 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 03:25:17.533223 (+ 2389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:25:17.534129 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:25:17.534141 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:25:17.534860 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:25:17.535659 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:17.535685 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:17.537300 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:17.537305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:17.538786 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:17.538804 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:17.539728 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:17.539736 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:17.540089 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:17.540898 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:25:17.540933 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:25:17.544000 (+ 3067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:25:17.547474 (+ 3474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:25:17.547487 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:25:17.547507 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:25:17.548149 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:25:17.548555 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:25:17.548560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:25:17.548564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:25:17.548852 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:25:17.549491 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:25:17.549501 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:25:17.550002 (+ 501us) client_negotiation.cc:769] Sending connection context
0924 03:25:17.550317 (+ 315us) client_negotiation.cc:240] Negotiation successful
0924 03:25:17.551059 (+ 742us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":353,"threads_started":1}
I20260924 03:25:17.554556 26300 heartbeater.cc:347] Connected to a master server at 127.10.12.190:39019
I20260924 03:25:17.555034 26300 heartbeater.cc:464] Registering TS with master...
I20260924 03:25:17.556092 26300 heartbeater.cc:511] Master 127.10.12.190:39019 requested a full tablet report, sending...
I20260924 03:25:17.558264 25843 ts_manager.cc:194] Registered new tserver with Master: cd9dfeb5e9404919b0c4d22a61a6518a (127.10.12.131:40007)
I20260924 03:25:17.559371 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:25:17.559506 25843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.131@KRBTEST.COM'} at 127.10.12.131:39171
I20260924 03:25:17.582772 26302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:17.565737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39350 (local address 127.10.12.190:39019)
0924 03:25:17.565943 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:17.565948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:17.566464 (+ 516us) server_negotiation.cc:407] Connection header received
0924 03:25:17.566756 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:17.566761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:17.566857 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:17.566965 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:25:17.568992 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:17.569896 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:17.571772 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:17.572370 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:17.572509 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:17.575587 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:25:17.575643 (+ 56us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:25:17.575647 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:25:17.575703 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:25:17.579074 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:17.580162 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:17.580168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:17.580170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:17.580381 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:17.580771 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:17.580774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:17.580776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:17.581208 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:25:17.581466 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:17.582022 (+ 556us) server_negotiation.cc:299] Negotiation successful
0924 03:25:17.582339 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:25:17.588598 25843 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:25:17.596763 25843 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:25:17.625636 26313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:17.613050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:44323 (local address 127.0.0.1:47626)
0924 03:25:17.614180 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:17.614249 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 03:25:17.614495 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:25:17.615945 (+ 1450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:25:17.615981 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:25:17.616016 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:25:17.616580 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:17.616599 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:17.619158 (+ 2559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:17.619164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:17.622169 (+ 3005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:17.622188 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:17.624255 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:17.624262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:17.624517 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:17.624651 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 03:25:17.624784 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 03:25:17.625089 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"mutex_wait_us":104,"thread_start_us":603,"threads_started":1}
I20260924 03:25:17.626607 26316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:17.614203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48488 (local address 127.10.12.129:42927)
0924 03:25:17.615433 (+ 1230us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:17.615442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:17.615849 (+ 407us) server_negotiation.cc:407] Connection header received
0924 03:25:17.615989 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:17.615997 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:17.616272 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:17.616505 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:25:17.617485 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:17.619090 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:17.622623 (+ 3533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:17.623870 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:17.624064 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:17.624376 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:17.625980 (+ 1604us) server_negotiation.cc:299] Negotiation successful
0924 03:25:17.626216 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1055,"thread_start_us":771,"threads_started":1}
I20260924 03:25:17.627759 26315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:17.613406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42927 (local address 127.0.0.1:48488)
0924 03:25:17.615560 (+ 2154us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:17.615580 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:25:17.615712 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:25:17.616773 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:25:17.616777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:25:17.616805 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:25:17.617269 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:17.617284 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:17.619247 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:17.619252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:17.622426 (+ 3174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:17.622440 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:17.624543 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:17.624550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:17.624841 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:17.624881 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 03:25:17.627095 (+ 2214us) client_negotiation.cc:240] Negotiation successful
0924 03:25:17.627295 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2033,"mutex_wait_us":809,"thread_start_us":1897,"threads_started":1}
I20260924 03:25:17.628208 26314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:17.613366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47626 (local address 127.10.12.130:44323)
0924 03:25:17.615148 (+ 1782us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:17.615168 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:17.615204 (+ 36us) server_negotiation.cc:407] Connection header received
0924 03:25:17.615342 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:17.615351 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:17.615686 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:17.616838 (+ 1152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:25:17.617105 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:17.618956 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:17.622431 (+ 3475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:17.624000 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:17.627123 (+ 3123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:17.627490 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:17.627593 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 03:25:17.627816 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1621,"thread_start_us":1473,"threads_started":1}
I20260924 03:25:17.631386 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:17.615679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:40007 (local address 127.0.0.1:47996)
0924 03:25:17.621010 (+ 5331us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:17.621037 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 03:25:17.621213 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:25:17.622134 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:25:17.622139 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:25:17.622174 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:25:17.622719 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:17.622733 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:17.624815 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:17.624820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:17.627215 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:17.627235 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:17.630418 (+ 3183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:17.630427 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:17.630641 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:17.630683 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 03:25:17.630784 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 03:25:17.630956 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5177,"mutex_wait_us":3,"thread_start_us":5703,"threads_started":1}
I20260924 03:25:17.632416 26318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:17.616108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47996 (local address 127.10.12.131:40007)
0924 03:25:17.620563 (+ 4455us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:17.620574 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:17.621383 (+ 809us) server_negotiation.cc:407] Connection header received
0924 03:25:17.621534 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:17.621544 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:17.621873 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:17.622123 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:25:17.622950 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:17.624600 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:17.627438 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:17.629793 (+ 2355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:17.629948 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:17.630219 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:17.631778 (+ 1559us) server_negotiation.cc:299] Negotiation successful
0924 03:25:17.632035 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4234,"thread_start_us":4016,"threads_started":1}
I20260924 03:25:17.633400 25967 tablet_service.cc:1467] Processing CreateTablet for tablet 45746e6e47e24c7d9130eec4606bb627 (DEFAULT_TABLE table=test-table [id=716ed071bf1540fa8d1a1127278b18a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:25:17.633395 26101 tablet_service.cc:1467] Processing CreateTablet for tablet 45746e6e47e24c7d9130eec4606bb627 (DEFAULT_TABLE table=test-table [id=716ed071bf1540fa8d1a1127278b18a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:25:17.635550 25967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45746e6e47e24c7d9130eec4606bb627. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:25:17.635653 26101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45746e6e47e24c7d9130eec4606bb627. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:25:17.637956 26235 tablet_service.cc:1467] Processing CreateTablet for tablet 45746e6e47e24c7d9130eec4606bb627 (DEFAULT_TABLE table=test-table [id=716ed071bf1540fa8d1a1127278b18a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:25:17.640417 26235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45746e6e47e24c7d9130eec4606bb627. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:25:17.654064 26319 tablet_bootstrap.cc:492] T 45746e6e47e24c7d9130eec4606bb627 P 4aa96ce3582e4c9abc4290645cd71562: Bootstrap starting.
I20260924 03:25:17.658342 26319 tablet_bootstrap.cc:654] T 45746e6e47e24c7d9130eec4606bb627 P 4aa96ce3582e4c9abc4290645cd71562: Neither blocks nor log segments found. Creating new log.
I20260924 03:25:17.659775 26319 log.cc:824] T 45746e6e47e24c7d9130eec4606bb627 P 4aa96ce3582e4c9abc4290645cd71562: Log is configured to *not* fsync() on all Append() calls
I20260924 03:25:17.661890 26321 tablet_bootstrap.cc:492] T 45746e6e47e24c7d9130eec4606bb627 P a5d08a1e9e764220811433108eb7dcbe: Bootstrap starting.
I20260924 03:25:17.663996 26319 tablet_bootstrap.cc:492] T 45746e6e47e24c7d9130eec4606bb627 P 4aa96ce3582e4c9abc4290645cd71562: No bootstrap required, opened a new log
I20260924 03:25:17.664433 26319 ts_tablet_manager.cc:1397] T 45746e6e47e24c7d9130eec4606bb627 P 4aa96ce3582e4c9abc4290645cd71562: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260924 03:25:17.666170 26322 tablet_bootstrap.cc:492] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a: Bootstrap starting.
I20260924 03:25:17.666795 26321 tablet_bootstrap.cc:654] T 45746e6e47e24c7d9130eec4606bb627 P a5d08a1e9e764220811433108eb7dcbe: Neither blocks nor log segments found. Creating new log.
I20260924 03:25:17.668447 26321 log.cc:824] T 45746e6e47e24c7d9130eec4606bb627 P a5d08a1e9e764220811433108eb7dcbe: Log is configured to *not* fsync() on all Append() calls
I20260924 03:25:17.670382 26322 tablet_bootstrap.cc:654] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a: Neither blocks nor log segments found. Creating new log.
I20260924 03:25:17.671839 26322 log.cc:824] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:25:17.673017 26321 tablet_bootstrap.cc:492] T 45746e6e47e24c7d9130eec4606bb627 P a5d08a1e9e764220811433108eb7dcbe: No bootstrap required, opened a new log
I20260924 03:25:17.673444 26321 ts_tablet_manager.cc:1397] T 45746e6e47e24c7d9130eec4606bb627 P a5d08a1e9e764220811433108eb7dcbe: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260924 03:25:17.676879 26319 raft_consensus.cc:348] T 45746e6e47e24c7d9130eec4606bb627 P 4aa96ce3582e4c9abc4290645cd71562 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d08a1e9e764220811433108eb7dcbe" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44323 } } peers { permanent_uuid: "4aa96ce3582e4c9abc4290645cd71562" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42927 } } peers { permanent_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40007 } }
I20260924 03:25:17.677647 26319 raft_consensus.cc:374] T 45746e6e47e24c7d9130eec4606bb627 P 4aa96ce3582e4c9abc4290645cd71562 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:25:17.677803 26319 raft_consensus.cc:729] T 45746e6e47e24c7d9130eec4606bb627 P 4aa96ce3582e4c9abc4290645cd71562 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aa96ce3582e4c9abc4290645cd71562, State: Initialized, Role: FOLLOWER
I20260924 03:25:17.678948 26319 consensus_queue.cc:260] T 45746e6e47e24c7d9130eec4606bb627 P 4aa96ce3582e4c9abc4290645cd71562 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d08a1e9e764220811433108eb7dcbe" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44323 } } peers { permanent_uuid: "4aa96ce3582e4c9abc4290645cd71562" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42927 } } peers { permanent_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40007 } }
I20260924 03:25:17.684880 26322 tablet_bootstrap.cc:492] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a: No bootstrap required, opened a new log
I20260924 03:25:17.685348 26319 ts_tablet_manager.cc:1428] T 45746e6e47e24c7d9130eec4606bb627 P 4aa96ce3582e4c9abc4290645cd71562: Time spent starting tablet: real 0.021s user 0.013s sys 0.004s
I20260924 03:25:17.685514 26322 ts_tablet_manager.cc:1397] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a: Time spent bootstrapping tablet: real 0.020s user 0.016s sys 0.000s
I20260924 03:25:17.686182 26321 raft_consensus.cc:348] T 45746e6e47e24c7d9130eec4606bb627 P a5d08a1e9e764220811433108eb7dcbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d08a1e9e764220811433108eb7dcbe" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44323 } } peers { permanent_uuid: "4aa96ce3582e4c9abc4290645cd71562" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42927 } } peers { permanent_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40007 } }
I20260924 03:25:17.686659 26321 raft_consensus.cc:374] T 45746e6e47e24c7d9130eec4606bb627 P a5d08a1e9e764220811433108eb7dcbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:25:17.686806 26321 raft_consensus.cc:729] T 45746e6e47e24c7d9130eec4606bb627 P a5d08a1e9e764220811433108eb7dcbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d08a1e9e764220811433108eb7dcbe, State: Initialized, Role: FOLLOWER
I20260924 03:25:17.687503 26321 consensus_queue.cc:260] T 45746e6e47e24c7d9130eec4606bb627 P a5d08a1e9e764220811433108eb7dcbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d08a1e9e764220811433108eb7dcbe" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44323 } } peers { permanent_uuid: "4aa96ce3582e4c9abc4290645cd71562" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42927 } } peers { permanent_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40007 } }
I20260924 03:25:17.688879 26166 heartbeater.cc:503] Master 127.10.12.190:39019 was elected leader, sending a full tablet report...
I20260924 03:25:17.689810 26321 ts_tablet_manager.cc:1428] T 45746e6e47e24c7d9130eec4606bb627 P a5d08a1e9e764220811433108eb7dcbe: Time spent starting tablet: real 0.016s user 0.009s sys 0.005s
I20260924 03:25:17.697408 26322 raft_consensus.cc:348] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d08a1e9e764220811433108eb7dcbe" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44323 } } peers { permanent_uuid: "4aa96ce3582e4c9abc4290645cd71562" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42927 } } peers { permanent_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40007 } }
I20260924 03:25:17.698011 26322 raft_consensus.cc:374] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:25:17.698114 26322 raft_consensus.cc:729] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd9dfeb5e9404919b0c4d22a61a6518a, State: Initialized, Role: FOLLOWER
I20260924 03:25:17.699236 26322 consensus_queue.cc:260] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d08a1e9e764220811433108eb7dcbe" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44323 } } peers { permanent_uuid: "4aa96ce3582e4c9abc4290645cd71562" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42927 } } peers { permanent_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40007 } }
I20260924 03:25:17.700842 26300 heartbeater.cc:503] Master 127.10.12.190:39019 was elected leader, sending a full tablet report...
I20260924 03:25:17.701718 26322 ts_tablet_manager.cc:1428] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
W20260924 03:25:17.777670 26301 tablet.cc:2369] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:25:17.838406 26033 tablet.cc:2369] T 45746e6e47e24c7d9130eec4606bb627 P 4aa96ce3582e4c9abc4290645cd71562: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:25:17.865598 26167 tablet.cc:2369] T 45746e6e47e24c7d9130eec4606bb627 P a5d08a1e9e764220811433108eb7dcbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:25:18.069875 26327 raft_consensus.cc:482] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:25:18.070329 26327 raft_consensus.cc:504] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d08a1e9e764220811433108eb7dcbe" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44323 } } peers { permanent_uuid: "4aa96ce3582e4c9abc4290645cd71562" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42927 } } peers { permanent_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40007 } }
I20260924 03:25:18.072547 26327 leader_election.cc:288] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5d08a1e9e764220811433108eb7dcbe (127.10.12.130:44323), 4aa96ce3582e4c9abc4290645cd71562 (127.10.12.129:42927)
I20260924 03:25:18.081184 26316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:18.073268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:52665 (local address 127.10.12.129:42927)
0924 03:25:18.073525 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:18.073531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:18.074197 (+ 666us) server_negotiation.cc:407] Connection header received
0924 03:25:18.074354 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:18.074360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:18.074445 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:18.074539 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:25:18.075239 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:18.076506 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:18.079016 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:18.079946 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:18.080161 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:18.080270 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:18.080709 (+ 439us) server_negotiation.cc:299] Negotiation successful
0924 03:25:18.080882 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 03:25:18.081276 26314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:18.073180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:48043 (local address 127.10.12.130:44323)
0924 03:25:18.073452 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:18.073461 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:18.073978 (+ 517us) server_negotiation.cc:407] Connection header received
0924 03:25:18.074226 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:18.074232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:18.074336 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:18.074462 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:25:18.075234 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:18.076506 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:18.079013 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:18.080038 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:18.080285 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:18.080384 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:18.080742 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 03:25:18.080943 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:25:18.081357 26329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:18.072931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:44323 (local address 127.10.12.131:48043)
0924 03:25:18.073772 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:18.073797 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 03:25:18.074037 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:25:18.074495 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:25:18.074501 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:25:18.074533 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:25:18.075006 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:18.075025 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:18.076726 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:18.076731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:18.078772 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:18.078788 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:18.080341 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:18.080347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:18.080561 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:18.080586 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 03:25:18.080681 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 03:25:18.080859 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":138,"thread_start_us":727,"threads_started":1}
I20260924 03:25:18.081513 26328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:18.072927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42927 (local address 127.10.12.131:52665)
0924 03:25:18.074062 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:18.074078 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:25:18.074197 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:25:18.074559 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:25:18.074563 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:25:18.074597 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:25:18.075096 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:18.075106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:18.076727 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:18.076732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:18.078760 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:18.078787 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:18.080218 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:18.080227 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:18.080482 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:18.080528 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 03:25:18.080648 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 03:25:18.080843 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"mutex_wait_us":60,"thread_start_us":443,"threads_started":1}
I20260924 03:25:18.082724 26121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45746e6e47e24c7d9130eec4606bb627" candidate_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d08a1e9e764220811433108eb7dcbe" is_pre_election: true
I20260924 03:25:18.082957 25987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45746e6e47e24c7d9130eec4606bb627" candidate_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aa96ce3582e4c9abc4290645cd71562" is_pre_election: true
I20260924 03:25:18.083544 25987 raft_consensus.cc:2454] T 45746e6e47e24c7d9130eec4606bb627 P 4aa96ce3582e4c9abc4290645cd71562 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd9dfeb5e9404919b0c4d22a61a6518a in term 0.
I20260924 03:25:18.083545 26121 raft_consensus.cc:2454] T 45746e6e47e24c7d9130eec4606bb627 P a5d08a1e9e764220811433108eb7dcbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd9dfeb5e9404919b0c4d22a61a6518a in term 0.
I20260924 03:25:18.084583 26190 leader_election.cc:302] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5d08a1e9e764220811433108eb7dcbe, cd9dfeb5e9404919b0c4d22a61a6518a; no voters:
I20260924 03:25:18.085266 26327 raft_consensus.cc:2781] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:25:18.085438 26327 raft_consensus.cc:482] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:25:18.085544 26327 raft_consensus.cc:3037] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:25:18.088896 26327 raft_consensus.cc:504] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d08a1e9e764220811433108eb7dcbe" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44323 } } peers { permanent_uuid: "4aa96ce3582e4c9abc4290645cd71562" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42927 } } peers { permanent_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40007 } }
I20260924 03:25:18.089712 26327 leader_election.cc:288] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a [CANDIDATE]: Term 1 election: Requested vote from peers a5d08a1e9e764220811433108eb7dcbe (127.10.12.130:44323), 4aa96ce3582e4c9abc4290645cd71562 (127.10.12.129:42927)
I20260924 03:25:18.090425 25987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45746e6e47e24c7d9130eec4606bb627" candidate_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aa96ce3582e4c9abc4290645cd71562"
I20260924 03:25:18.090471 26121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45746e6e47e24c7d9130eec4606bb627" candidate_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d08a1e9e764220811433108eb7dcbe"
I20260924 03:25:18.090685 25987 raft_consensus.cc:3037] T 45746e6e47e24c7d9130eec4606bb627 P 4aa96ce3582e4c9abc4290645cd71562 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:25:18.090735 26121 raft_consensus.cc:3037] T 45746e6e47e24c7d9130eec4606bb627 P a5d08a1e9e764220811433108eb7dcbe [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:25:18.093523 26121 raft_consensus.cc:2454] T 45746e6e47e24c7d9130eec4606bb627 P a5d08a1e9e764220811433108eb7dcbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd9dfeb5e9404919b0c4d22a61a6518a in term 1.
I20260924 03:25:18.093523 25987 raft_consensus.cc:2454] T 45746e6e47e24c7d9130eec4606bb627 P 4aa96ce3582e4c9abc4290645cd71562 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd9dfeb5e9404919b0c4d22a61a6518a in term 1.
I20260924 03:25:18.094265 26188 leader_election.cc:302] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa96ce3582e4c9abc4290645cd71562, cd9dfeb5e9404919b0c4d22a61a6518a; no voters:
I20260924 03:25:18.094671 26327 raft_consensus.cc:2781] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:25:18.095208 26327 raft_consensus.cc:686] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a [term 1 LEADER]: Becoming Leader. State: Replica: cd9dfeb5e9404919b0c4d22a61a6518a, State: Running, Role: LEADER
I20260924 03:25:18.095822 26327 consensus_queue.cc:237] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d08a1e9e764220811433108eb7dcbe" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44323 } } peers { permanent_uuid: "4aa96ce3582e4c9abc4290645cd71562" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42927 } } peers { permanent_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40007 } }
I20260924 03:25:18.102180 25842 catalog_manager.cc:5495] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a reported cstate change: term changed from 0 to 1, leader changed from <none> to cd9dfeb5e9404919b0c4d22a61a6518a (127.10.12.131). New cstate: current_term: 1 leader_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d08a1e9e764220811433108eb7dcbe" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa96ce3582e4c9abc4290645cd71562" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40007 } health_report { overall_health: HEALTHY } } }
I20260924 03:25:18.219853 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:18.210940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48010 (local address 127.10.12.131:40007)
0924 03:25:18.211710 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:18.211717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:18.211744 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:25:18.211941 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:18.211946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:18.212109 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:18.212615 (+ 506us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:25:18.213112 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:18.213964 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:18.216357 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:18.217085 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:18.217280 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:18.217955 (+ 675us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:25:18.218864 (+ 909us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:25:18.219051 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:18.219188 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 03:25:18.219449 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":335,"threads_started":1}
I20260924 03:25:18.232489 26121 raft_consensus.cc:1261] T 45746e6e47e24c7d9130eec4606bb627 P a5d08a1e9e764220811433108eb7dcbe [term 1 FOLLOWER]: Refusing update from remote peer cd9dfeb5e9404919b0c4d22a61a6518a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:25:18.232489 25987 raft_consensus.cc:1261] T 45746e6e47e24c7d9130eec4606bb627 P 4aa96ce3582e4c9abc4290645cd71562 [term 1 FOLLOWER]: Refusing update from remote peer cd9dfeb5e9404919b0c4d22a61a6518a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:25:18.233887 26327 consensus_queue.cc:1035] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa96ce3582e4c9abc4290645cd71562" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:25:18.234351 26330 consensus_queue.cc:1035] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d08a1e9e764220811433108eb7dcbe" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:25:18.250739 26338 mvcc.cc:205] Tried to move back new op lower bound from 7332742423464382464 to 7332742422925729792. Current Snapshot: MvccSnapshot[applied={T|T < 7332742423464382464}]
I20260924 03:25:18.266505 25843 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:25:18.266956 25843 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:25:18.271742 25843 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef: Sending DeleteTablet for 3 replicas of tablet 45746e6e47e24c7d9130eec4606bb627
I20260924 03:25:18.273545 25967 tablet_service.cc:1514] Processing DeleteTablet for tablet 45746e6e47e24c7d9130eec4606bb627 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:25:18 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:48488
I20260924 03:25:18.273535 26101 tablet_service.cc:1514] Processing DeleteTablet for tablet 45746e6e47e24c7d9130eec4606bb627 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:25:18 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:47626
I20260924 03:25:18.273818 26235 tablet_service.cc:1514] Processing DeleteTablet for tablet 45746e6e47e24c7d9130eec4606bb627 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:25:18 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:47996
I20260924 03:25:18.274811 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 25904
I20260924 03:25:18.287063 26347 tablet_replica.cc:330] T 45746e6e47e24c7d9130eec4606bb627 P 4aa96ce3582e4c9abc4290645cd71562: stopping tablet replica
I20260924 03:25:18.288075 26347 raft_consensus.cc:2229] T 45746e6e47e24c7d9130eec4606bb627 P 4aa96ce3582e4c9abc4290645cd71562 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:25:18.289146 26347 raft_consensus.cc:2258] T 45746e6e47e24c7d9130eec4606bb627 P 4aa96ce3582e4c9abc4290645cd71562 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:25:18.290488 26348 tablet_replica.cc:330] T 45746e6e47e24c7d9130eec4606bb627 P a5d08a1e9e764220811433108eb7dcbe: stopping tablet replica
I20260924 03:25:18.293006 26347 ts_tablet_manager.cc:1905] T 45746e6e47e24c7d9130eec4606bb627 P 4aa96ce3582e4c9abc4290645cd71562: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:25:18.293987 26348 raft_consensus.cc:2229] T 45746e6e47e24c7d9130eec4606bb627 P a5d08a1e9e764220811433108eb7dcbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:25:18.295078 26348 raft_consensus.cc:2258] T 45746e6e47e24c7d9130eec4606bb627 P a5d08a1e9e764220811433108eb7dcbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:25:18.295344 26349 tablet_replica.cc:330] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a: stopping tablet replica
I20260924 03:25:18.296061 26349 raft_consensus.cc:2229] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:25:18.297295 26349 raft_consensus.cc:2258] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:25:18.299033 26348 ts_tablet_manager.cc:1905] T 45746e6e47e24c7d9130eec4606bb627 P a5d08a1e9e764220811433108eb7dcbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:25:18.305063 26349 ts_tablet_manager.cc:1905] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:25:18.305619 26347 ts_tablet_manager.cc:1918] T 45746e6e47e24c7d9130eec4606bb627 P 4aa96ce3582e4c9abc4290645cd71562: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:25:18.305864 26347 log.cc:1196] T 45746e6e47e24c7d9130eec4606bb627 P 4aa96ce3582e4c9abc4290645cd71562: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/wal/wals/45746e6e47e24c7d9130eec4606bb627
I20260924 03:25:18.306480 26347 ts_tablet_manager.cc:1939] T 45746e6e47e24c7d9130eec4606bb627 P 4aa96ce3582e4c9abc4290645cd71562: Deleting consensus metadata
I20260924 03:25:18.309051 26348 ts_tablet_manager.cc:1918] T 45746e6e47e24c7d9130eec4606bb627 P a5d08a1e9e764220811433108eb7dcbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:25:18.309216 25827 catalog_manager.cc:4841] TS 4aa96ce3582e4c9abc4290645cd71562 (127.10.12.129:42927): tablet 45746e6e47e24c7d9130eec4606bb627 (table test-table [id=716ed071bf1540fa8d1a1127278b18a6]) successfully deleted
I20260924 03:25:18.309576 26348 log.cc:1196] T 45746e6e47e24c7d9130eec4606bb627 P a5d08a1e9e764220811433108eb7dcbe: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/45746e6e47e24c7d9130eec4606bb627
I20260924 03:25:18.310321 26348 ts_tablet_manager.cc:1939] T 45746e6e47e24c7d9130eec4606bb627 P a5d08a1e9e764220811433108eb7dcbe: Deleting consensus metadata
I20260924 03:25:18.312963 25829 catalog_manager.cc:4841] TS a5d08a1e9e764220811433108eb7dcbe (127.10.12.130:44323): tablet 45746e6e47e24c7d9130eec4606bb627 (table test-table [id=716ed071bf1540fa8d1a1127278b18a6]) successfully deleted
I20260924 03:25:18.314821 26349 ts_tablet_manager.cc:1918] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:25:18.315035 26349 log.cc:1196] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/45746e6e47e24c7d9130eec4606bb627
I20260924 03:25:18.315620 26349 ts_tablet_manager.cc:1939] T 45746e6e47e24c7d9130eec4606bb627 P cd9dfeb5e9404919b0c4d22a61a6518a: Deleting consensus metadata
I20260924 03:25:18.317811 25827 catalog_manager.cc:4841] TS cd9dfeb5e9404919b0c4d22a61a6518a (127.10.12.131:40007): tablet 45746e6e47e24c7d9130eec4606bb627 (table test-table [id=716ed071bf1540fa8d1a1127278b18a6]) successfully deleted
Sep 24 03:25:18 dist-test-slave-00qs krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.129@KRBTEST.COM
I20260924 03:25:18.330108 26316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:18.276453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48492 (local address 127.10.12.129:42927)
0924 03:25:18.276696 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:18.276705 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:18.291164 (+ 14459us) server_negotiation.cc:407] Connection header received
0924 03:25:18.291572 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:18.291581 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:18.291708 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:18.291799 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:25:18.298184 (+ 6385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:18.299608 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:18.309879 (+ 10271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:18.310738 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:18.310886 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:18.322375 (+ 11489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:25:18.322402 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:25:18.322424 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:25:18.322485 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:25:18.325984 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:18.327312 (+ 1328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:18.327323 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:18.327326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:18.327533 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:18.328121 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:18.328127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:18.328131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:18.328682 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:25:18.328998 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:18.329438 (+ 440us) server_negotiation.cc:299] Negotiation successful
0924 03:25:18.329758 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 03:25:18.330922 26027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:25:18.433130 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 25904
I20260924 03:25:18.459578 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 26038
Sep 24 03:25:18 dist-test-slave-00qs krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.130@KRBTEST.COM
I20260924 03:25:18.479259 26314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:18.460761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47642 (local address 127.10.12.130:44323)
0924 03:25:18.460950 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:18.460956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:18.460979 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:25:18.461224 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:18.461230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:18.461316 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:18.461479 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:25:18.463289 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:18.464297 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:18.465850 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:18.466340 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:18.466476 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:18.472035 (+ 5559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:25:18.472059 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:25:18.472103 (+ 44us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:25:18.472162 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:25:18.475429 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:18.476439 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:18.476452 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:18.476456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:18.476698 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:18.477207 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:18.477213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:18.477216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:18.477835 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:25:18.478266 (+ 431us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:18.478521 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 03:25:18.478816 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 03:25:18.480232 26161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:25:18.597940 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 26038
I20260924 03:25:18.615232 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 26172
Sep 24 03:25:18 dist-test-slave-00qs krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.131@KRBTEST.COM
I20260924 03:25:18.641710 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:18.616312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48022 (local address 127.10.12.131:40007)
0924 03:25:18.616579 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:18.616586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:18.616635 (+ 49us) server_negotiation.cc:407] Connection header received
0924 03:25:18.616866 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:18.616872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:18.617001 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:18.617144 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:25:18.619043 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:18.620591 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:18.622479 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:18.623259 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:18.623460 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:18.630701 (+ 7241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:25:18.630736 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:25:18.630765 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:25:18.630861 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:25:18.637192 (+ 6331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:18.638511 (+ 1319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:18.638532 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:18.638537 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:18.638784 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:18.639324 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:18.639334 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:18.639339 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:18.640058 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:25:18.640440 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:18.640854 (+ 414us) server_negotiation.cc:299] Negotiation successful
0924 03:25:18.641224 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 03:25:18.642823 26295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:25:18.789893 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 26172
I20260924 03:25:18.808573 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 25811
I20260924 03:25:18.810186 25873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:25:18.942312 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 25811
I20260924 03:25:38.966657 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:39019
--webserver_interface=127.10.12.190
--webserver_port=32933
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:39019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:25:39.415361 26355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:25:39.415840 26355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:25:39.416011 26355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:25:39.425293 26355 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 03:25:39.425488 26355 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 03:25:39.425554 26355 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 03:25:39.425637 26355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:25:39.425729 26355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:25:39.425839 26355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:25:39.425904 26355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:25:39.425985 26355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:25:39.438268 26355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:39019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:39019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=32933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:25:39.440243 26355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:25:39.442005 26355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:25:39.476989 26364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:25:39.476964 26362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:25:39.476939 26361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:25:39.477730 26355 server_base.cc:1034] running on GCE node
I20260924 03:25:39.479326 26355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:25:39.482023 26355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:25:39.483376 26355 hybrid_clock.cc:648] HybridClock initialized: now 1790220339483290 us; error 81 us; skew 500 ppm
I20260924 03:25:39.484025 26355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:25:39 dist-test-slave-00qs krb5kdc[25798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220339, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:25:39.497138 26355 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:25:39.499850 26355 webserver.cc:460] Webserver started at http://127.10.12.190:32933/ using document root <none> and password file <none>
I20260924 03:25:39.501047 26355 fs_manager.cc:359] Metadata directory not provided
I20260924 03:25:39.501224 26355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:25:39.511144 26355 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.001s sys 0.004s
I20260924 03:25:39.516060 26371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:25:39.518318 26355 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260924 03:25:39.518690 26355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "3f1b56edf4c14316aae772d84db31fef"
format_stamp: "Formatted at 2026-09-24 03:25:15 on dist-test-slave-00qs"
server_key: "3e9c374c4bb2685da4f70d477b0154a1"
server_key_iv: "d0da3792bb788181fdb9e952f2f4fd4f"
server_key_version: "encryptionkey@0"
I20260924 03:25:39.520349 26355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:25:39.555177 26355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:25:39.561271 26355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:25:39.561620 26355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:25:39.584654 26355 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:39019
I20260924 03:25:39.584677 26422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:39019 every 8 connection(s)
I20260924 03:25:39.586848 26355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:25:39.594646 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 26355
I20260924 03:25:39.595839 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:42927
--local_ip_for_outbound_sockets=127.10.12.129
--tserver_master_addrs=127.10.12.190:39019
--webserver_port=44953
--webserver_interface=127.10.12.129
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:25:39.599308 26423 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:25:39.606061 26423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef: Bootstrap starting.
I20260924 03:25:39.632728 26423 log.cc:824] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef: Log is configured to *not* fsync() on all Append() calls
I20260924 03:25:39.651286 26423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:25:39.652042 26423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef: Bootstrap complete.
I20260924 03:25:39.663854 26423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b56edf4c14316aae772d84db31fef" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39019 } }
I20260924 03:25:39.665153 26423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1b56edf4c14316aae772d84db31fef, State: Initialized, Role: FOLLOWER
I20260924 03:25:39.666231 26423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b56edf4c14316aae772d84db31fef" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39019 } }
I20260924 03:25:39.666786 26423 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:25:39.666972 26423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:25:39.667270 26423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:25:39.673228 26423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b56edf4c14316aae772d84db31fef" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39019 } }
I20260924 03:25:39.674201 26423 leader_election.cc:302] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f1b56edf4c14316aae772d84db31fef; no voters:
I20260924 03:25:39.675027 26423 leader_election.cc:288] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 03:25:39.675215 26427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [term 2 FOLLOWER]: Leader election won for term 2
I20260924 03:25:39.676519 26427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [term 2 LEADER]: Becoming Leader. State: Replica: 3f1b56edf4c14316aae772d84db31fef, State: Running, Role: LEADER
I20260924 03:25:39.677356 26427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b56edf4c14316aae772d84db31fef" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39019 } }
I20260924 03:25:39.677994 26423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:25:39.681272 26428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3f1b56edf4c14316aae772d84db31fef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b56edf4c14316aae772d84db31fef" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39019 } } }
I20260924 03:25:39.681268 26429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f1b56edf4c14316aae772d84db31fef. Latest consensus state: current_term: 2 leader_uuid: "3f1b56edf4c14316aae772d84db31fef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b56edf4c14316aae772d84db31fef" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39019 } } }
I20260924 03:25:39.681862 26428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [sys.catalog]: This master's current role is: LEADER
I20260924 03:25:39.681875 26429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [sys.catalog]: This master's current role is: LEADER
I20260924 03:25:39.685994 26432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:25:39.705873 26432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:25:39.707894 26432 catalog_manager.cc:1259] Loaded cluster ID: 682e1b919ea04febbbdd19dc3c5e5caa
I20260924 03:25:39.708163 26432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:25:39.713917 26432 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:25:39.713853 26444 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 03:25:39.716815 26432 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef: Loaded TSK: 0
I20260924 03:25:39.752543 26432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef: Generated new TSK 1
I20260924 03:25:39.753171 26432 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 03:25:40.096915 26425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:25:40.097386 26425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:25:40.097553 26425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:25:40.108089 26425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:25:40.108291 26425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:25:40.108477 26425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:25:40.122419 26425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:42927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=44953
--tserver_master_addrs=127.10.12.190:39019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:25:40.124698 26425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:25:40.126756 26425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:25:40.146803 26450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:25:40.146685 26451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:25:40.147305 26453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:25:40.149065 26425 server_base.cc:1034] running on GCE node
I20260924 03:25:40.151053 26425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:25:40.152896 26425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:25:40.156404 26425 hybrid_clock.cc:648] HybridClock initialized: now 1790220340156318 us; error 80 us; skew 500 ppm
I20260924 03:25:40.157027 26425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:25:40 dist-test-slave-00qs krb5kdc[25798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220340, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:25:40.162918 26425 init.cc:377] Logged in from keytab as kudu/127.10.12.129@KRBTEST.COM (short username kudu)
I20260924 03:25:40.165686 26425 webserver.cc:460] Webserver started at http://127.10.12.129:44953/ using document root <none> and password file <none>
I20260924 03:25:40.166759 26425 fs_manager.cc:359] Metadata directory not provided
I20260924 03:25:40.166957 26425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:25:40.175203 26425 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:25:40.179817 26460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:25:40.182024 26425 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260924 03:25:40.182349 26425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "4aa96ce3582e4c9abc4290645cd71562"
format_stamp: "Formatted at 2026-09-24 03:25:16 on dist-test-slave-00qs"
server_key: "342f08ff6e5e065deeefb9bf431aad93"
server_key_iv: "34a07bfef0433181c695ab07c3687584"
server_key_version: "encryptionkey@0"
I20260924 03:25:40.183693 26425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:25:40.206175 26425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:25:40.213029 26425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:25:40.213552 26425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:25:40.214941 26425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:25:40.216852 26425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:25:40.217003 26425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:25:40.217133 26425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:25:40.217214 26425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:25:40.255129 26425 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:42927
I20260924 03:25:40.255218 26572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:42927 every 8 connection(s)
I20260924 03:25:40.257078 26425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:25:40.258611 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 26425
I20260924 03:25:40.259485 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:44323
--local_ip_for_outbound_sockets=127.10.12.130
--tserver_master_addrs=127.10.12.190:39019
--webserver_port=44653
--webserver_interface=127.10.12.130
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:25:40 dist-test-slave-00qs krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220340, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:25:40.293269 26576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:40.261730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.129:34223)
0924 03:25:40.262860 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:40.262935 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0924 03:25:40.265137 (+ 2202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:25:40.267583 (+ 2446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:25:40.267593 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:25:40.268440 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:25:40.269208 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:40.269245 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:40.271377 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:40.271383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:40.272956 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:40.272977 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:40.274695 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:40.274702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:40.274958 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:40.276069 (+ 1111us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:25:40.276103 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:25:40.279785 (+ 3682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:25:40.286554 (+ 6769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:25:40.286567 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:25:40.286587 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:25:40.287397 (+ 810us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:25:40.288391 (+ 994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:25:40.288397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:25:40.288401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:25:40.288827 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:25:40.290227 (+ 1400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:25:40.290239 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:25:40.290833 (+ 594us) client_negotiation.cc:769] Sending connection context
0924 03:25:40.291158 (+ 325us) client_negotiation.cc:240] Negotiation successful
0924 03:25:40.291765 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":413,"threads_started":1}
I20260924 03:25:40.293813 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:40.261266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:34223 (local address 127.10.12.190:39019)
0924 03:25:40.262226 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:40.262251 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:40.263139 (+ 888us) server_negotiation.cc:407] Connection header received
0924 03:25:40.266092 (+ 2953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:40.266116 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:40.266808 (+ 692us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:40.267378 (+ 570us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:25:40.269543 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:40.271132 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:40.273176 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:40.274350 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:40.274604 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:40.280354 (+ 5750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:25:40.280369 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:25:40.280397 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:25:40.280481 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:25:40.286135 (+ 5654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:40.287699 (+ 1564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:40.287717 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:40.287728 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:40.288172 (+ 444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:40.289002 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:40.289008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:40.289012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:40.290026 (+ 1014us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:25:40.290343 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:40.291538 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0924 03:25:40.292218 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":376,"threads_started":1}
I20260924 03:25:40.297726 26573 heartbeater.cc:347] Connected to a master server at 127.10.12.190:39019
I20260924 03:25:40.298226 26573 heartbeater.cc:464] Registering TS with master...
I20260924 03:25:40.299465 26573 heartbeater.cc:511] Master 127.10.12.190:39019 requested a full tablet report, sending...
I20260924 03:25:40.302793 26388 ts_manager.cc:194] Registered new tserver with Master: 4aa96ce3582e4c9abc4290645cd71562 (127.10.12.129:42927)
I20260924 03:25:40.304723 26388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.129@KRBTEST.COM'} at 127.10.12.129:34223
W20260924 03:25:40.697394 26577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:25:40.697824 26577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:25:40.697974 26577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:25:40.707357 26577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:25:40.707551 26577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:25:40.707753 26577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:25:40.721088 26577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:44323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=44653
--tserver_master_addrs=127.10.12.190:39019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:25:40.722918 26577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:25:40.724921 26577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:25:40.746029 26582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:25:40.747004 26585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:25:40.747668 26583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:25:40.749711 26577 server_base.cc:1034] running on GCE node
I20260924 03:25:40.750840 26577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:25:40.752617 26577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:25:40.754070 26577 hybrid_clock.cc:648] HybridClock initialized: now 1790220340753993 us; error 82 us; skew 500 ppm
I20260924 03:25:40.754603 26577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:25:40 dist-test-slave-00qs krb5kdc[25798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220340, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:25:40.758971 26577 init.cc:377] Logged in from keytab as kudu/127.10.12.130@KRBTEST.COM (short username kudu)
I20260924 03:25:40.761385 26577 webserver.cc:460] Webserver started at http://127.10.12.130:44653/ using document root <none> and password file <none>
I20260924 03:25:40.762506 26577 fs_manager.cc:359] Metadata directory not provided
I20260924 03:25:40.762701 26577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:25:40.771442 26577 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.003s
I20260924 03:25:40.776926 26592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:25:40.779596 26577 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.002s
I20260924 03:25:40.780077 26577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "a5d08a1e9e764220811433108eb7dcbe"
format_stamp: "Formatted at 2026-09-24 03:25:16 on dist-test-slave-00qs"
server_key: "329f2709cd00c5ba8317072c59970e7d"
server_key_iv: "fa5778828f770eebe6b1105185f46267"
server_key_version: "encryptionkey@0"
I20260924 03:25:40.781817 26577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:25:40.828147 26577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:25:40.834576 26577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:25:40.834980 26577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:25:40.836515 26577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:25:40.838533 26577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:25:40.838690 26577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:25:40.838847 26577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:25:40.838927 26577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:25:40.881032 26577 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:44323
I20260924 03:25:40.881466 26704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:44323 every 8 connection(s)
I20260924 03:25:40.883601 26577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:25:40.892323 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 26577
I20260924 03:25:40.893178 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:40007
--local_ip_for_outbound_sockets=127.10.12.131
--tserver_master_addrs=127.10.12.190:39019
--webserver_port=35227
--webserver_interface=127.10.12.131
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:25:40 dist-test-slave-00qs krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220340, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:25:40.919703 26707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:40.888694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:43183 (local address 127.10.12.190:39019)
0924 03:25:40.889369 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:40.889374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:40.890885 (+ 1511us) server_negotiation.cc:407] Connection header received
0924 03:25:40.895057 (+ 4172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:40.895064 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:40.895254 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:40.895557 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:25:40.898419 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:40.899428 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:40.901928 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:40.902809 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:40.903229 (+ 420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:40.910040 (+ 6811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:25:40.910058 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:25:40.910064 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:25:40.910149 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:25:40.914584 (+ 4435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:40.916263 (+ 1679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:40.916270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:40.916273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:40.916502 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:40.917153 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:40.917158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:40.917160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:40.917550 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:25:40.917746 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:40.918847 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0924 03:25:40.919231 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":338,"threads_started":1}
I20260924 03:25:40.921146 26708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:40.889464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.130:43183)
0924 03:25:40.890618 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:40.890684 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 03:25:40.894419 (+ 3735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:25:40.896018 (+ 1599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:25:40.896037 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:25:40.896914 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:25:40.898039 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:40.898073 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:40.899858 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:40.899865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:40.901616 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:40.901647 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:40.903449 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:40.903460 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:40.903852 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:40.904855 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:25:40.904890 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:25:40.909456 (+ 4566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:25:40.914996 (+ 5540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:25:40.915010 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:25:40.915040 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:25:40.916004 (+ 964us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:25:40.916668 (+ 664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:25:40.916675 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:25:40.916679 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:25:40.916987 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:25:40.917729 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:25:40.917740 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:25:40.918288 (+ 548us) client_negotiation.cc:769] Sending connection context
0924 03:25:40.918622 (+ 334us) client_negotiation.cc:240] Negotiation successful
0924 03:25:40.919461 (+ 839us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":387,"threads_started":1}
I20260924 03:25:40.923784 26705 heartbeater.cc:347] Connected to a master server at 127.10.12.190:39019
I20260924 03:25:40.924322 26705 heartbeater.cc:464] Registering TS with master...
I20260924 03:25:40.925438 26705 heartbeater.cc:511] Master 127.10.12.190:39019 requested a full tablet report, sending...
I20260924 03:25:40.927829 26388 ts_manager.cc:194] Registered new tserver with Master: a5d08a1e9e764220811433108eb7dcbe (127.10.12.130:44323)
I20260924 03:25:40.928921 26388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.130@KRBTEST.COM'} at 127.10.12.130:43183
I20260924 03:25:41.309682 26573 heartbeater.cc:503] Master 127.10.12.190:39019 was elected leader, sending a full tablet report...
W20260924 03:25:41.343242 26709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:25:41.343658 26709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:25:41.343801 26709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:25:41.352456 26709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:25:41.352643 26709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:25:41.352818 26709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:25:41.366094 26709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:40007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=35227
--tserver_master_addrs=127.10.12.190:39019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:25:41.368278 26709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:25:41.370373 26709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:25:41.388427 26715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:25:41.388413 26717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:25:41.389299 26714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:25:41.389709 26709 server_base.cc:1034] running on GCE node
I20260924 03:25:41.390586 26709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:25:41.392042 26709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:25:41.393288 26709 hybrid_clock.cc:648] HybridClock initialized: now 1790220341393213 us; error 74 us; skew 500 ppm
I20260924 03:25:41.393698 26709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:25:41 dist-test-slave-00qs krb5kdc[25798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220341, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:25:41.397557 26709 init.cc:377] Logged in from keytab as kudu/127.10.12.131@KRBTEST.COM (short username kudu)
I20260924 03:25:41.399547 26709 webserver.cc:460] Webserver started at http://127.10.12.131:35227/ using document root <none> and password file <none>
I20260924 03:25:41.400518 26709 fs_manager.cc:359] Metadata directory not provided
I20260924 03:25:41.400660 26709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:25:41.408161 26709 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 03:25:41.412483 26724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:25:41.414528 26709 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 03:25:41.414778 26709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "cd9dfeb5e9404919b0c4d22a61a6518a"
format_stamp: "Formatted at 2026-09-24 03:25:17 on dist-test-slave-00qs"
server_key: "bba68c1301e128a321885e4ecdcae5fd"
server_key_iv: "a2797c99994ccd33becb02efd70ac660"
server_key_version: "encryptionkey@0"
I20260924 03:25:41.416153 26709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:25:41.442287 26709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:25:41.448961 26709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:25:41.449389 26709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:25:41.451083 26709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:25:41.453182 26709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:25:41.453333 26709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:25:41.453477 26709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:25:41.453567 26709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:25:41.494342 26709 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:40007
I20260924 03:25:41.494390 26836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:40007 every 8 connection(s)
I20260924 03:25:41.496291 26709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:25:41.502262 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 26709
Sep 24 03:25:41 dist-test-slave-00qs krb5kdc[25798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220341, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:25:41.525055 26842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:41.503983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33092 (local address 127.10.12.190:39019)
0924 03:25:41.504686 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:41.504691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:41.504710 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:25:41.504766 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:41.504770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:41.504854 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:41.504980 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:25:41.507614 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.509117 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.511802 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.512603 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.512788 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:41.517052 (+ 4264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:25:41.517060 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:25:41.517064 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:25:41.517122 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:25:41.521294 (+ 4172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:41.522523 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:41.522529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:41.522531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:41.522678 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:41.523123 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:41.523126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:41.523128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:41.523580 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:25:41.523823 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:41.524307 (+ 484us) server_negotiation.cc:299] Negotiation successful
0924 03:25:41.524650 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":576,"thread_start_us":403,"threads_started":1}
I20260924 03:25:41.526459 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:41.499600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:44641 (local address 127.10.12.190:39019)
0924 03:25:41.500191 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:41.500197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:41.501354 (+ 1157us) server_negotiation.cc:407] Connection header received
0924 03:25:41.504177 (+ 2823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:41.504184 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:41.504346 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:41.504471 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:25:41.507889 (+ 3418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.509195 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.512283 (+ 3088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.512950 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.513085 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:41.518493 (+ 5408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:25:41.518506 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:25:41.518511 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:25:41.518583 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:25:41.522160 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:41.523251 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:41.523256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:41.523259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:41.523434 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:41.524139 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:41.524144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:41.524146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:41.524495 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:25:41.524708 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:41.525712 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0924 03:25:41.525957 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":447,"thread_start_us":269,"threads_started":1}
I20260924 03:25:41.527928 26840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:41.500252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.131:44641)
0924 03:25:41.501115 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:41.501171 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 03:25:41.503522 (+ 2351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:25:41.505289 (+ 1767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:25:41.505309 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:25:41.506381 (+ 1072us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:25:41.507639 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:41.507671 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.509413 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.509419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:41.512052 (+ 2633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:41.512072 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.513326 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.513336 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:41.513763 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:41.514567 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:25:41.514613 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:25:41.518053 (+ 3440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:25:41.522339 (+ 4286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:25:41.522351 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:25:41.522374 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:25:41.523121 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:25:41.523555 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:25:41.523559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:25:41.523561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:25:41.523983 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:25:41.524703 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:25:41.524713 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:25:41.525267 (+ 554us) client_negotiation.cc:769] Sending connection context
0924 03:25:41.525629 (+ 362us) client_negotiation.cc:240] Negotiation successful
0924 03:25:41.526300 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":310,"threads_started":1}
I20260924 03:25:41.529829 26837 heartbeater.cc:347] Connected to a master server at 127.10.12.190:39019
I20260924 03:25:41.530205 26837 heartbeater.cc:464] Registering TS with master...
I20260924 03:25:41.531210 26837 heartbeater.cc:511] Master 127.10.12.190:39019 requested a full tablet report, sending...
I20260924 03:25:41.533255 26388 ts_manager.cc:194] Registered new tserver with Master: cd9dfeb5e9404919b0c4d22a61a6518a (127.10.12.131:40007)
I20260924 03:25:41.534284 26388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.131@KRBTEST.COM'} at 127.10.12.131:44641
I20260924 03:25:41.541446 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:25:41.549597 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:41.542634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33094 (local address 127.10.12.190:39019)
0924 03:25:41.542887 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:41.542892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:41.543472 (+ 580us) server_negotiation.cc:407] Connection header received
0924 03:25:41.543730 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:41.543734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:41.543946 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:41.544149 (+ 203us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:25:41.544681 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.545536 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.547338 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.547813 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.547940 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:41.548597 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:25:41.548898 (+ 301us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 03:25:41.549317 (+ 419us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33094: token expired
Metrics: {"server-negotiator.queue_time_us":115}
W20260924 03:25:41.549737 26839 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33094: token expired
W20260924 03:25:41.550477 26304 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.10.12.190:39019)
I20260924 03:25:41.566769 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:41.551064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33104 (local address 127.10.12.190:39019)
0924 03:25:41.551253 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:41.551257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:41.551275 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:25:41.551558 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:41.551562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:41.551644 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:41.551744 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:25:41.553582 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.554501 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.556257 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.556743 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.556907 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:41.560183 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:25:41.560190 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:25:41.560193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:25:41.560248 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:25:41.563465 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:41.564485 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:41.564490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:41.564493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:41.564726 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:41.565128 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:41.565131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:41.565134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:41.565561 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:25:41.565896 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:41.566175 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 03:25:41.566407 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 03:25:41.571496 26388 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:25:41.580570 26388 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:25:41.629081 26853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:41.615191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51380 (local address 127.10.12.131:40007)
0924 03:25:41.620091 (+ 4900us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:41.620100 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:41.620136 (+ 36us) server_negotiation.cc:407] Connection header received
0924 03:25:41.620239 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:41.620246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:41.620514 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:41.620742 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:25:41.621318 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.623017 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.626171 (+ 3154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.627548 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.627715 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:41.628071 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:41.628435 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 03:25:41.628644 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4647,"thread_start_us":4442,"threads_started":1}
I20260924 03:25:41.629139 26852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:41.615076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:40007 (local address 127.0.0.1:51380)
0924 03:25:41.617202 (+ 2126us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:41.617448 (+ 246us) client_negotiation.cc:174] Beginning negotiation
0924 03:25:41.617845 (+ 397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:25:41.620732 (+ 2887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:25:41.620739 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:25:41.620769 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:25:41.621074 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:41.621084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.623203 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.623208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:41.625935 (+ 2727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:41.625955 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.627826 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.627834 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:41.628135 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:41.628198 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 03:25:41.628340 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 03:25:41.628635 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1959,"mutex_wait_us":58,"thread_start_us":1767,"threads_started":1}
I20260924 03:25:41.629637 26849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:41.600959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42927 (local address 127.0.0.1:38116)
0924 03:25:41.613357 (+ 12398us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:41.613422 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 03:25:41.613721 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:25:41.618549 (+ 4828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:25:41.618561 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:25:41.618607 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:25:41.619165 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:41.619181 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.623598 (+ 4417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.623604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:41.625754 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:41.625791 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.628502 (+ 2711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.628509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:41.628727 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:41.628763 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 03:25:41.628861 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 03:25:41.629033 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11976,"mutex_wait_us":7,"thread_start_us":12993,"threads_started":1}
I20260924 03:25:41.629914 26850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:41.601306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38116 (local address 127.10.12.129:42927)
0924 03:25:41.616516 (+ 15210us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:41.616530 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:41.616579 (+ 49us) server_negotiation.cc:407] Connection header received
0924 03:25:41.616742 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:41.616755 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:41.617166 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:41.617523 (+ 357us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:25:41.621300 (+ 3777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.623446 (+ 2146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.626122 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.627853 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.628041 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:41.628312 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:41.629321 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0924 03:25:41.629548 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14931,"thread_start_us":14717,"threads_started":1}
I20260924 03:25:41.631469 26848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:41.601209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:44323 (local address 127.0.0.1:39232)
0924 03:25:41.614302 (+ 13093us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:41.614325 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:25:41.614473 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:25:41.620134 (+ 5661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:25:41.620139 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:25:41.620174 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:25:41.620565 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:41.620588 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.622849 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.622855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:41.625944 (+ 3089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:41.625955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.630579 (+ 4624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.630586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:41.630791 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:41.630826 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 03:25:41.630910 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 03:25:41.631081 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12936,"mutex_wait_us":92,"thread_start_us":11554,"threads_started":1}
I20260924 03:25:41.632231 26851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:41.601534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39232 (local address 127.10.12.130:44323)
0924 03:25:41.614453 (+ 12919us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:41.614464 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:41.614508 (+ 44us) server_negotiation.cc:407] Connection header received
0924 03:25:41.614654 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:41.614661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:41.615019 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:41.615290 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:25:41.620845 (+ 5555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.622670 (+ 1825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.626162 (+ 3492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.629600 (+ 3438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.629763 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:41.630017 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:41.631743 (+ 1726us) server_negotiation.cc:299] Negotiation successful
0924 03:25:41.631931 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12685,"thread_start_us":13996,"threads_started":1}
I20260924 03:25:41.634903 26772 tablet_service.cc:1467] Processing CreateTablet for tablet e7e8cbb9d9c840d184b81018372d401f (DEFAULT_TABLE table=test-table [id=9718dacc9ab14ac9a9f7751d1d0cced8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:25:41.635526 26640 tablet_service.cc:1467] Processing CreateTablet for tablet e7e8cbb9d9c840d184b81018372d401f (DEFAULT_TABLE table=test-table [id=9718dacc9ab14ac9a9f7751d1d0cced8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:25:41.636631 26772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7e8cbb9d9c840d184b81018372d401f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:25:41.637079 26640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7e8cbb9d9c840d184b81018372d401f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:25:41.639295 26508 tablet_service.cc:1467] Processing CreateTablet for tablet e7e8cbb9d9c840d184b81018372d401f (DEFAULT_TABLE table=test-table [id=9718dacc9ab14ac9a9f7751d1d0cced8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:25:41.641332 26508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7e8cbb9d9c840d184b81018372d401f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:25:41.656605 26854 tablet_bootstrap.cc:492] T e7e8cbb9d9c840d184b81018372d401f P cd9dfeb5e9404919b0c4d22a61a6518a: Bootstrap starting.
I20260924 03:25:41.662938 26854 tablet_bootstrap.cc:654] T e7e8cbb9d9c840d184b81018372d401f P cd9dfeb5e9404919b0c4d22a61a6518a: Neither blocks nor log segments found. Creating new log.
I20260924 03:25:41.664695 26854 log.cc:824] T e7e8cbb9d9c840d184b81018372d401f P cd9dfeb5e9404919b0c4d22a61a6518a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:25:41.664947 26855 tablet_bootstrap.cc:492] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe: Bootstrap starting.
I20260924 03:25:41.668720 26857 tablet_bootstrap.cc:492] T e7e8cbb9d9c840d184b81018372d401f P 4aa96ce3582e4c9abc4290645cd71562: Bootstrap starting.
I20260924 03:25:41.669661 26854 tablet_bootstrap.cc:492] T e7e8cbb9d9c840d184b81018372d401f P cd9dfeb5e9404919b0c4d22a61a6518a: No bootstrap required, opened a new log
I20260924 03:25:41.670174 26854 ts_tablet_manager.cc:1397] T e7e8cbb9d9c840d184b81018372d401f P cd9dfeb5e9404919b0c4d22a61a6518a: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260924 03:25:41.670207 26855 tablet_bootstrap.cc:654] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe: Neither blocks nor log segments found. Creating new log.
I20260924 03:25:41.671706 26855 log.cc:824] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe: Log is configured to *not* fsync() on all Append() calls
I20260924 03:25:41.674574 26857 tablet_bootstrap.cc:654] T e7e8cbb9d9c840d184b81018372d401f P 4aa96ce3582e4c9abc4290645cd71562: Neither blocks nor log segments found. Creating new log.
I20260924 03:25:41.676491 26857 log.cc:824] T e7e8cbb9d9c840d184b81018372d401f P 4aa96ce3582e4c9abc4290645cd71562: Log is configured to *not* fsync() on all Append() calls
I20260924 03:25:41.677040 26855 tablet_bootstrap.cc:492] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe: No bootstrap required, opened a new log
I20260924 03:25:41.677503 26855 ts_tablet_manager.cc:1397] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260924 03:25:41.681806 26857 tablet_bootstrap.cc:492] T e7e8cbb9d9c840d184b81018372d401f P 4aa96ce3582e4c9abc4290645cd71562: No bootstrap required, opened a new log
I20260924 03:25:41.682296 26857 ts_tablet_manager.cc:1397] T e7e8cbb9d9c840d184b81018372d401f P 4aa96ce3582e4c9abc4290645cd71562: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.002s
I20260924 03:25:41.683596 26854 raft_consensus.cc:348] T e7e8cbb9d9c840d184b81018372d401f P cd9dfeb5e9404919b0c4d22a61a6518a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa96ce3582e4c9abc4290645cd71562" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42927 } } peers { permanent_uuid: "a5d08a1e9e764220811433108eb7dcbe" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44323 } } peers { permanent_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40007 } }
I20260924 03:25:41.684306 26854 raft_consensus.cc:374] T e7e8cbb9d9c840d184b81018372d401f P cd9dfeb5e9404919b0c4d22a61a6518a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:25:41.684451 26854 raft_consensus.cc:729] T e7e8cbb9d9c840d184b81018372d401f P cd9dfeb5e9404919b0c4d22a61a6518a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd9dfeb5e9404919b0c4d22a61a6518a, State: Initialized, Role: FOLLOWER
I20260924 03:25:41.685390 26854 consensus_queue.cc:260] T e7e8cbb9d9c840d184b81018372d401f P cd9dfeb5e9404919b0c4d22a61a6518a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa96ce3582e4c9abc4290645cd71562" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42927 } } peers { permanent_uuid: "a5d08a1e9e764220811433108eb7dcbe" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44323 } } peers { permanent_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40007 } }
I20260924 03:25:41.687577 26855 raft_consensus.cc:348] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa96ce3582e4c9abc4290645cd71562" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42927 } } peers { permanent_uuid: "a5d08a1e9e764220811433108eb7dcbe" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44323 } } peers { permanent_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40007 } }
I20260924 03:25:41.688611 26837 heartbeater.cc:503] Master 127.10.12.190:39019 was elected leader, sending a full tablet report...
I20260924 03:25:41.688655 26855 raft_consensus.cc:374] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:25:41.688793 26855 raft_consensus.cc:729] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d08a1e9e764220811433108eb7dcbe, State: Initialized, Role: FOLLOWER
I20260924 03:25:41.689850 26855 consensus_queue.cc:260] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa96ce3582e4c9abc4290645cd71562" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42927 } } peers { permanent_uuid: "a5d08a1e9e764220811433108eb7dcbe" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44323 } } peers { permanent_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40007 } }
I20260924 03:25:41.690497 26854 ts_tablet_manager.cc:1428] T e7e8cbb9d9c840d184b81018372d401f P cd9dfeb5e9404919b0c4d22a61a6518a: Time spent starting tablet: real 0.020s user 0.010s sys 0.007s
I20260924 03:25:41.692641 26705 heartbeater.cc:503] Master 127.10.12.190:39019 was elected leader, sending a full tablet report...
I20260924 03:25:41.693516 26855 ts_tablet_manager.cc:1428] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260924 03:25:41.696190 26857 raft_consensus.cc:348] T e7e8cbb9d9c840d184b81018372d401f P 4aa96ce3582e4c9abc4290645cd71562 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa96ce3582e4c9abc4290645cd71562" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42927 } } peers { permanent_uuid: "a5d08a1e9e764220811433108eb7dcbe" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44323 } } peers { permanent_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40007 } }
I20260924 03:25:41.696832 26857 raft_consensus.cc:374] T e7e8cbb9d9c840d184b81018372d401f P 4aa96ce3582e4c9abc4290645cd71562 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:25:41.697005 26857 raft_consensus.cc:729] T e7e8cbb9d9c840d184b81018372d401f P 4aa96ce3582e4c9abc4290645cd71562 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aa96ce3582e4c9abc4290645cd71562, State: Initialized, Role: FOLLOWER
I20260924 03:25:41.698041 26857 consensus_queue.cc:260] T e7e8cbb9d9c840d184b81018372d401f P 4aa96ce3582e4c9abc4290645cd71562 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa96ce3582e4c9abc4290645cd71562" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42927 } } peers { permanent_uuid: "a5d08a1e9e764220811433108eb7dcbe" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44323 } } peers { permanent_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40007 } }
I20260924 03:25:41.700505 26857 ts_tablet_manager.cc:1428] T e7e8cbb9d9c840d184b81018372d401f P 4aa96ce3582e4c9abc4290645cd71562: Time spent starting tablet: real 0.018s user 0.017s sys 0.002s
W20260924 03:25:41.748418 26838 tablet.cc:2369] T e7e8cbb9d9c840d184b81018372d401f P cd9dfeb5e9404919b0c4d22a61a6518a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:25:41.760667 26574 tablet.cc:2369] T e7e8cbb9d9c840d184b81018372d401f P 4aa96ce3582e4c9abc4290645cd71562: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:25:41.830428 26861 raft_consensus.cc:482] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:25:41.830855 26861 raft_consensus.cc:504] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa96ce3582e4c9abc4290645cd71562" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42927 } } peers { permanent_uuid: "a5d08a1e9e764220811433108eb7dcbe" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44323 } } peers { permanent_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40007 } }
I20260924 03:25:41.833214 26861 leader_election.cc:288] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aa96ce3582e4c9abc4290645cd71562 (127.10.12.129:42927), cd9dfeb5e9404919b0c4d22a61a6518a (127.10.12.131:40007)
I20260924 03:25:41.841614 26850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:41.833794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:60501 (local address 127.10.12.129:42927)
0924 03:25:41.834015 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:41.834021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:41.834492 (+ 471us) server_negotiation.cc:407] Connection header received
0924 03:25:41.834780 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:41.834786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:41.834886 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:41.835016 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:25:41.835801 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.836942 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.839278 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.840444 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.840604 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:41.840677 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:41.841137 (+ 460us) server_negotiation.cc:299] Negotiation successful
0924 03:25:41.841312 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 03:25:41.841660 26863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:41.833605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42927 (local address 127.10.12.130:60501)
0924 03:25:41.834378 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:41.834401 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:25:41.834580 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:25:41.835101 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:25:41.835105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:25:41.835161 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:25:41.835596 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:41.835610 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.837098 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.837105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:41.839028 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:41.839044 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.840718 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.840725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:41.840928 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:41.840967 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 03:25:41.841077 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 03:25:41.841262 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":411,"threads_started":1}
I20260924 03:25:41.842108 26853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:41.834711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:55121 (local address 127.10.12.131:40007)
0924 03:25:41.834991 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:41.834999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:41.835646 (+ 647us) server_negotiation.cc:407] Connection header received
0924 03:25:41.835812 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:41.835816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:41.835887 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:41.835997 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:25:41.836736 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.837744 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.839865 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.840971 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.841149 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:41.841244 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:41.841686 (+ 442us) server_negotiation.cc:299] Negotiation successful
0924 03:25:41.841850 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:25:41.842131 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:41.834527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:40007 (local address 127.10.12.130:55121)
0924 03:25:41.835504 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:41.835522 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:25:41.835671 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:25:41.836040 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:25:41.836045 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:25:41.836072 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:25:41.836485 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:41.836499 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.837932 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.837936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:41.839648 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:25:41.839664 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.841223 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.841229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:25:41.841481 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:41.841514 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 03:25:41.841630 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 03:25:41.841785 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"thread_start_us":531,"threads_started":1}
I20260924 03:25:41.843298 26528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e8cbb9d9c840d184b81018372d401f" candidate_uuid: "a5d08a1e9e764220811433108eb7dcbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aa96ce3582e4c9abc4290645cd71562" is_pre_election: true
I20260924 03:25:41.843611 26792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e8cbb9d9c840d184b81018372d401f" candidate_uuid: "a5d08a1e9e764220811433108eb7dcbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" is_pre_election: true
I20260924 03:25:41.844202 26528 raft_consensus.cc:2454] T e7e8cbb9d9c840d184b81018372d401f P 4aa96ce3582e4c9abc4290645cd71562 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d08a1e9e764220811433108eb7dcbe in term 0.
I20260924 03:25:41.844203 26792 raft_consensus.cc:2454] T e7e8cbb9d9c840d184b81018372d401f P cd9dfeb5e9404919b0c4d22a61a6518a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d08a1e9e764220811433108eb7dcbe in term 0.
I20260924 03:25:41.845173 26593 leader_election.cc:302] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa96ce3582e4c9abc4290645cd71562, a5d08a1e9e764220811433108eb7dcbe; no voters:
I20260924 03:25:41.845669 26861 raft_consensus.cc:2781] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:25:41.845853 26861 raft_consensus.cc:482] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:25:41.845973 26861 raft_consensus.cc:3037] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:25:41.848860 26861 raft_consensus.cc:504] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa96ce3582e4c9abc4290645cd71562" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42927 } } peers { permanent_uuid: "a5d08a1e9e764220811433108eb7dcbe" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44323 } } peers { permanent_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40007 } }
I20260924 03:25:41.849675 26861 leader_election.cc:288] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe [CANDIDATE]: Term 1 election: Requested vote from peers 4aa96ce3582e4c9abc4290645cd71562 (127.10.12.129:42927), cd9dfeb5e9404919b0c4d22a61a6518a (127.10.12.131:40007)
I20260924 03:25:41.850656 26792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e8cbb9d9c840d184b81018372d401f" candidate_uuid: "a5d08a1e9e764220811433108eb7dcbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a"
I20260924 03:25:41.850656 26528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e8cbb9d9c840d184b81018372d401f" candidate_uuid: "a5d08a1e9e764220811433108eb7dcbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aa96ce3582e4c9abc4290645cd71562"
I20260924 03:25:41.850991 26528 raft_consensus.cc:3037] T e7e8cbb9d9c840d184b81018372d401f P 4aa96ce3582e4c9abc4290645cd71562 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:25:41.850981 26792 raft_consensus.cc:3037] T e7e8cbb9d9c840d184b81018372d401f P cd9dfeb5e9404919b0c4d22a61a6518a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:25:41.854961 26528 raft_consensus.cc:2454] T e7e8cbb9d9c840d184b81018372d401f P 4aa96ce3582e4c9abc4290645cd71562 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d08a1e9e764220811433108eb7dcbe in term 1.
I20260924 03:25:41.854961 26792 raft_consensus.cc:2454] T e7e8cbb9d9c840d184b81018372d401f P cd9dfeb5e9404919b0c4d22a61a6518a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d08a1e9e764220811433108eb7dcbe in term 1.
I20260924 03:25:41.855720 26593 leader_election.cc:302] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa96ce3582e4c9abc4290645cd71562, a5d08a1e9e764220811433108eb7dcbe; no voters:
I20260924 03:25:41.856256 26861 raft_consensus.cc:2781] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:25:41.856889 26861 raft_consensus.cc:686] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe [term 1 LEADER]: Becoming Leader. State: Replica: a5d08a1e9e764220811433108eb7dcbe, State: Running, Role: LEADER
I20260924 03:25:41.857527 26861 consensus_queue.cc:237] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa96ce3582e4c9abc4290645cd71562" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42927 } } peers { permanent_uuid: "a5d08a1e9e764220811433108eb7dcbe" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44323 } } peers { permanent_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40007 } }
I20260924 03:25:41.863783 26388 catalog_manager.cc:5495] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe reported cstate change: term changed from 0 to 1, leader changed from <none> to a5d08a1e9e764220811433108eb7dcbe (127.10.12.130). New cstate: current_term: 1 leader_uuid: "a5d08a1e9e764220811433108eb7dcbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aa96ce3582e4c9abc4290645cd71562" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5d08a1e9e764220811433108eb7dcbe" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40007 } health_report { overall_health: UNKNOWN } } }
W20260924 03:25:41.887375 26706 tablet.cc:2369] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:25:41.937814 26851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:41.931350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39238 (local address 127.10.12.130:44323)
0924 03:25:41.931605 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:41.931612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:41.931699 (+ 87us) server_negotiation.cc:407] Connection header received
0924 03:25:41.931932 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:41.931937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:41.932092 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:41.932269 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:25:41.932690 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.933474 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.935073 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:41.935541 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:41.935677 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:41.936359 (+ 682us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:25:41.937058 (+ 699us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:25:41.937221 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:41.937333 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 03:25:41.937450 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 03:25:41.950373 26528 raft_consensus.cc:1261] T e7e8cbb9d9c840d184b81018372d401f P 4aa96ce3582e4c9abc4290645cd71562 [term 1 FOLLOWER]: Refusing update from remote peer a5d08a1e9e764220811433108eb7dcbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:25:41.950373 26792 raft_consensus.cc:1261] T e7e8cbb9d9c840d184b81018372d401f P cd9dfeb5e9404919b0c4d22a61a6518a [term 1 FOLLOWER]: Refusing update from remote peer a5d08a1e9e764220811433108eb7dcbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:25:41.952011 26861 consensus_queue.cc:1035] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa96ce3582e4c9abc4290645cd71562" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:25:41.952595 26865 consensus_queue.cc:1035] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd9dfeb5e9404919b0c4d22a61a6518a" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 40007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:25:41.970932 26869 mvcc.cc:205] Tried to move back new op lower bound from 7332742520612229120 to 7332742520254050304. Current Snapshot: MvccSnapshot[applied={T|T < 7332742520612229120}]
I20260924 03:25:41.993737 26387 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:25:41.994153 26387 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:25:41.999317 26387 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef: Sending DeleteTablet for 3 replicas of tablet e7e8cbb9d9c840d184b81018372d401f
I20260924 03:25:42.001259 26508 tablet_service.cc:1514] Processing DeleteTablet for tablet e7e8cbb9d9c840d184b81018372d401f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:25:41 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:38116
I20260924 03:25:42.001276 26640 tablet_service.cc:1514] Processing DeleteTablet for tablet e7e8cbb9d9c840d184b81018372d401f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:25:41 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:39232
I20260924 03:25:42.001902 26772 tablet_service.cc:1514] Processing DeleteTablet for tablet e7e8cbb9d9c840d184b81018372d401f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:25:41 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:51380
I20260924 03:25:42.004405 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 26425
I20260924 03:25:42.013737 26878 tablet_replica.cc:330] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe: stopping tablet replica
I20260924 03:25:42.014039 26880 tablet_replica.cc:330] T e7e8cbb9d9c840d184b81018372d401f P cd9dfeb5e9404919b0c4d22a61a6518a: stopping tablet replica
I20260924 03:25:42.014472 26878 raft_consensus.cc:2229] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:25:42.014675 26880 raft_consensus.cc:2229] T e7e8cbb9d9c840d184b81018372d401f P cd9dfeb5e9404919b0c4d22a61a6518a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:25:42.015524 26880 raft_consensus.cc:2258] T e7e8cbb9d9c840d184b81018372d401f P cd9dfeb5e9404919b0c4d22a61a6518a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:25:42.015491 26878 raft_consensus.cc:2258] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:25:42.016671 26879 tablet_replica.cc:330] T e7e8cbb9d9c840d184b81018372d401f P 4aa96ce3582e4c9abc4290645cd71562: stopping tablet replica
I20260924 03:25:42.017406 26879 raft_consensus.cc:2229] T e7e8cbb9d9c840d184b81018372d401f P 4aa96ce3582e4c9abc4290645cd71562 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:25:42.018169 26879 raft_consensus.cc:2258] T e7e8cbb9d9c840d184b81018372d401f P 4aa96ce3582e4c9abc4290645cd71562 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:25:42.019181 26880 ts_tablet_manager.cc:1905] T e7e8cbb9d9c840d184b81018372d401f P cd9dfeb5e9404919b0c4d22a61a6518a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:25:42.019306 26878 ts_tablet_manager.cc:1905] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:25:42.029019 26880 ts_tablet_manager.cc:1918] T e7e8cbb9d9c840d184b81018372d401f P cd9dfeb5e9404919b0c4d22a61a6518a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:25:42.029212 26878 ts_tablet_manager.cc:1918] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:25:42.029273 26880 log.cc:1196] T e7e8cbb9d9c840d184b81018372d401f P cd9dfeb5e9404919b0c4d22a61a6518a: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/e7e8cbb9d9c840d184b81018372d401f
I20260924 03:25:42.029417 26878 log.cc:1196] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/e7e8cbb9d9c840d184b81018372d401f
I20260924 03:25:42.029908 26880 ts_tablet_manager.cc:1939] T e7e8cbb9d9c840d184b81018372d401f P cd9dfeb5e9404919b0c4d22a61a6518a: Deleting consensus metadata
I20260924 03:25:42.030052 26878 ts_tablet_manager.cc:1939] T e7e8cbb9d9c840d184b81018372d401f P a5d08a1e9e764220811433108eb7dcbe: Deleting consensus metadata
I20260924 03:25:42.032428 26879 ts_tablet_manager.cc:1905] T e7e8cbb9d9c840d184b81018372d401f P 4aa96ce3582e4c9abc4290645cd71562: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:25:42.032486 26374 catalog_manager.cc:4841] TS a5d08a1e9e764220811433108eb7dcbe (127.10.12.130:44323): tablet e7e8cbb9d9c840d184b81018372d401f (table test-table [id=9718dacc9ab14ac9a9f7751d1d0cced8]) successfully deleted
I20260924 03:25:42.032496 26372 catalog_manager.cc:4841] TS cd9dfeb5e9404919b0c4d22a61a6518a (127.10.12.131:40007): tablet e7e8cbb9d9c840d184b81018372d401f (table test-table [id=9718dacc9ab14ac9a9f7751d1d0cced8]) successfully deleted
I20260924 03:25:42.035661 26850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:42.005881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38124 (local address 127.10.12.129:42927)
0924 03:25:42.006101 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:42.006108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:42.006133 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:25:42.006217 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:42.006224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:42.006326 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:42.006454 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:25:42.009409 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:42.010858 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:42.014704 (+ 3846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:42.015469 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:42.015616 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:42.022704 (+ 7088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:25:42.022725 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:25:42.022760 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:25:42.022832 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:25:42.027653 (+ 4821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:42.031178 (+ 3525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:42.031205 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:42.031209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:42.031486 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:42.033206 (+ 1720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:42.033214 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:42.033218 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:42.033934 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:25:42.034411 (+ 477us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:42.034820 (+ 409us) server_negotiation.cc:299] Negotiation successful
0924 03:25:42.035159 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 03:25:42.036582 26568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:25:42.173532 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 26425
I20260924 03:25:42.191608 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 26577
W20260924 03:25:42.192044 26372 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.12.129:42927 (error 108)
I20260924 03:25:42.196875 26883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:42.195425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42927 (local address 127.0.0.1:38134)
0924 03:25:42.196412 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:42.196599 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:42927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":675,"threads_started":1}
W20260924 03:25:42.197310 26372 catalog_manager.cc:4584] TS 4aa96ce3582e4c9abc4290645cd71562 (127.10.12.129:42927): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e7e8cbb9d9c840d184b81018372d401f: Network error: Client connection negotiation failed: client connection to 127.10.12.129:42927: connect: Connection refused (error 111)
I20260924 03:25:42.214879 26851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:42.192665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39246 (local address 127.10.12.130:44323)
0924 03:25:42.192905 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:42.192914 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:42.193270 (+ 356us) server_negotiation.cc:407] Connection header received
0924 03:25:42.193365 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:42.193370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:42.193463 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:42.193560 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:25:42.195165 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:42.196537 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:42.198273 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:42.198777 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:42.198917 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:42.206942 (+ 8025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:25:42.206960 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:25:42.206981 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:25:42.207042 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:25:42.210930 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:42.212042 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:42.212053 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:42.212056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:42.212276 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:42.212745 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:42.212750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:42.212752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:42.213248 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:25:42.213661 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:42.214086 (+ 425us) server_negotiation.cc:299] Negotiation successful
0924 03:25:42.214424 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 03:25:42.215843 26700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:25:42.251869 26883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:42.251237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42927 (local address 127.0.0.1:38140)
0924 03:25:42.251522 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:42.251648 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:42927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260924 03:25:42.253345 26883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:42.252879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42927 (local address 127.0.0.1:38156)
0924 03:25:42.253054 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:42.253150 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:42927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 03:25:42.314400 26883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:42.313780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42927 (local address 127.0.0.1:38160)
0924 03:25:42.314076 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:42.314199 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:42927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260924 03:25:42.315733 26883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:42.315240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42927 (local address 127.0.0.1:38176)
0924 03:25:42.315409 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:42.315499 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:42927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 03:25:42.350528 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 26577
I20260924 03:25:42.368619 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 26709
I20260924 03:25:42.385488 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:42.369589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51388 (local address 127.10.12.131:40007)
0924 03:25:42.370219 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0924 03:25:42.370224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:25:42.370240 (+ 16us) server_negotiation.cc:407] Connection header received
0924 03:25:42.370320 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:25:42.370324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:25:42.370401 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:25:42.370508 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:25:42.372186 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:42.372986 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:42.374362 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:25:42.374798 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:25:42.374923 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:25:42.378392 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:25:42.378416 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:25:42.378437 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:25:42.378490 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:25:42.381866 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:42.382871 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:42.382881 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:42.382884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:42.383069 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:25:42.383524 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:25:42.383529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:25:42.383531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:25:42.384038 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:25:42.384377 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0924 03:25:42.384745 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 03:25:42.385027 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":329,"threads_started":1}
I20260924 03:25:42.386296 26832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:25:42.397008 26883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:42.396325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42927 (local address 127.0.0.1:38188)
0924 03:25:42.396677 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:42.396803 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:42927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260924 03:25:42.398361 26883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:42.397846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42927 (local address 127.0.0.1:38204)
0924 03:25:42.398119 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:42.398204 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:42927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260924 03:25:42.488247 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 26709
I20260924 03:25:42.504658 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 26355
I20260924 03:25:42.506160 26418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:25:42.642526 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 26355
I20260924 03:25:42.642575 26883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:25:42.641912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:42927 (local address 127.0.0.1:38214)
0924 03:25:42.642217 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 03:25:42.642354 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:42927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 03:26:02.665514 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:39019
--webserver_interface=127.10.12.190
--webserver_port=32933
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:39019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:26:03.109344 26888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:03.109922 26888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:03.110095 26888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:03.121357 26888 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 03:26:03.121553 26888 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 03:26:03.121627 26888 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 03:26:03.121709 26888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:26:03.121797 26888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:03.121905 26888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:03.121971 26888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:26:03.122049 26888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:26:03.134572 26888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:39019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:39019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=32933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:03.136480 26888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:03.138306 26888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:03.153409 26893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:03.153381 26894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:03.153683 26888 server_base.cc:1034] running on GCE node
W20260924 03:26:03.153319 26896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:03.154975 26888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:03.156817 26888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:03.158061 26888 hybrid_clock.cc:648] HybridClock initialized: now 1790220363157973 us; error 56 us; skew 500 ppm
I20260924 03:26:03.158493 26888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:26:03 dist-test-slave-00qs krb5kdc[25798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220363, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:26:03.163069 26888 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:26:03.165383 26888 webserver.cc:460] Webserver started at http://127.10.12.190:32933/ using document root <none> and password file <none>
I20260924 03:26:03.166407 26888 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:03.166548 26888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:03.174081 26888 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 03:26:03.177932 26903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:03.179682 26888 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260924 03:26:03.179885 26888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "3f1b56edf4c14316aae772d84db31fef"
format_stamp: "Formatted at 2026-09-24 03:25:15 on dist-test-slave-00qs"
server_key: "3e9c374c4bb2685da4f70d477b0154a1"
server_key_iv: "d0da3792bb788181fdb9e952f2f4fd4f"
server_key_version: "encryptionkey@0"
I20260924 03:26:03.181126 26888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:03.198546 26888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:03.204036 26888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:03.204385 26888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:03.225807 26888 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:39019
I20260924 03:26:03.225826 26954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:39019 every 8 connection(s)
I20260924 03:26:03.227790 26888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:26:03.229938 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 26888
I20260924 03:26:03.230968 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:42927
--local_ip_for_outbound_sockets=127.10.12.129
--tserver_master_addrs=127.10.12.190:39019
--webserver_port=44953
--webserver_interface=127.10.12.129
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:26:03.237426 26955 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:26:03.242050 26955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef: Bootstrap starting.
I20260924 03:26:03.279788 26955 log.cc:824] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:03.307745 26955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:26:03.308776 26955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef: Bootstrap complete.
I20260924 03:26:03.325277 26955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b56edf4c14316aae772d84db31fef" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39019 } }
I20260924 03:26:03.326833 26955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1b56edf4c14316aae772d84db31fef, State: Initialized, Role: FOLLOWER
I20260924 03:26:03.327797 26955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b56edf4c14316aae772d84db31fef" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39019 } }
I20260924 03:26:03.328315 26955 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:26:03.328504 26955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:26:03.328749 26955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [term 2 FOLLOWER]: Advancing to term 3
I20260924 03:26:03.333930 26955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b56edf4c14316aae772d84db31fef" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39019 } }
I20260924 03:26:03.334640 26955 leader_election.cc:302] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f1b56edf4c14316aae772d84db31fef; no voters:
I20260924 03:26:03.335454 26955 leader_election.cc:288] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 03:26:03.335698 26959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [term 3 FOLLOWER]: Leader election won for term 3
I20260924 03:26:03.336990 26959 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [term 3 LEADER]: Becoming Leader. State: Replica: 3f1b56edf4c14316aae772d84db31fef, State: Running, Role: LEADER
I20260924 03:26:03.337775 26959 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b56edf4c14316aae772d84db31fef" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39019 } }
I20260924 03:26:03.338158 26955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:26:03.341066 26960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3f1b56edf4c14316aae772d84db31fef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b56edf4c14316aae772d84db31fef" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39019 } } }
I20260924 03:26:03.341037 26961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f1b56edf4c14316aae772d84db31fef. Latest consensus state: current_term: 3 leader_uuid: "3f1b56edf4c14316aae772d84db31fef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b56edf4c14316aae772d84db31fef" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 39019 } } }
I20260924 03:26:03.341501 26960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:03.341553 26961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:03.356907 26966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:26:03.371707 26966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:26:03.374814 26966 catalog_manager.cc:1259] Loaded cluster ID: 682e1b919ea04febbbdd19dc3c5e5caa
I20260924 03:26:03.375172 26966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:26:03.381690 26966 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:26:03.384953 26966 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef: Loaded TSK: 0
I20260924 03:26:03.385900 26966 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3f1b56edf4c14316aae772d84db31fef: Loaded TSK: 1
F20260924 03:26:03.387856 26966 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f836fd4b9ad google::LogMessage::Fail() at ??:0
@ 0x7f836fd4e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f836fd4b4de google::LogMessage::Flush() at ??:0
@ 0x7f836fd4ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8373aaf7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f8373a9e73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f838577a1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f8385779783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f83858052a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f83858050c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f838584f5e9 std::function<>::operator()() at ??:0
@ 0x7f838577dcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f838577bc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8385803610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f8385803439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f8372997345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f837299a91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f837299a799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f8372975eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8375c4f6db start_thread at ??:0
@ 0x7f836ebe561f clone at ??:0
W20260924 03:26:03.777143 26957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:03.777601 26957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:03.777725 26957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:03.786504 26957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:03.786706 26957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:03.786847 26957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:26:03.798633 26957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:42927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=44953
--tserver_master_addrs=127.10.12.190:39019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:03.800420 26957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:03.802203 26957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:03.818799 27005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:03.819065 27002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:03.818852 27003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:03.819796 26957 server_base.cc:1034] running on GCE node
I20260924 03:26:03.820835 26957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:03.822079 26957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:03.823323 26957 hybrid_clock.cc:648] HybridClock initialized: now 1790220363823250 us; error 69 us; skew 500 ppm
I20260924 03:26:03.823772 26957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:26:03 dist-test-slave-00qs krb5kdc[25798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220363, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:26:03.827975 26957 init.cc:377] Logged in from keytab as kudu/127.10.12.129@KRBTEST.COM (short username kudu)
I20260924 03:26:03.830265 26957 webserver.cc:460] Webserver started at http://127.10.12.129:44953/ using document root <none> and password file <none>
I20260924 03:26:03.831292 26957 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:03.831442 26957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:03.839047 26957 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:26:03.843503 27012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:03.845966 26957 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:26:03.846203 26957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "4aa96ce3582e4c9abc4290645cd71562"
format_stamp: "Formatted at 2026-09-24 03:25:16 on dist-test-slave-00qs"
server_key: "342f08ff6e5e065deeefb9bf431aad93"
server_key_iv: "34a07bfef0433181c695ab07c3687584"
server_key_version: "encryptionkey@0"
I20260924 03:26:03.847927 26957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:03.869009 26957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:03.874930 26957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:03.875289 26957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:03.876680 26957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:26:03.879467 27019 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260924 03:26:03.886991 27019 ts_tablet_manager.cc:1852] T e7e8cbb9d9c840d184b81018372d401f P 4aa96ce3582e4c9abc4290645cd71562: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 03:26:03.888326 27019 ts_tablet_manager.cc:1905] T e7e8cbb9d9c840d184b81018372d401f P 4aa96ce3582e4c9abc4290645cd71562: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:26:03.898710 27019 ts_tablet_manager.cc:1918] T e7e8cbb9d9c840d184b81018372d401f P 4aa96ce3582e4c9abc4290645cd71562: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:26:03.898981 27019 log.cc:1196] T e7e8cbb9d9c840d184b81018372d401f P 4aa96ce3582e4c9abc4290645cd71562: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/wal/wals/e7e8cbb9d9c840d184b81018372d401f
I20260924 03:26:03.899559 27019 ts_tablet_manager.cc:1939] T e7e8cbb9d9c840d184b81018372d401f P 4aa96ce3582e4c9abc4290645cd71562: Deleting consensus metadata
I20260924 03:26:03.900094 26957 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 03:26:03.900246 26957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.021s user 0.001s sys 0.000s
I20260924 03:26:03.900408 26957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:26:03.900492 26957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:03.940011 26957 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:42927
I20260924 03:26:03.940135 27125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:42927 every 8 connection(s)
I20260924 03:26:03.941846 26957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:26:03.943938 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 26957
I20260924 03:26:03.944761 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:44323
--local_ip_for_outbound_sockets=127.10.12.130
--tserver_master_addrs=127.10.12.190:39019
--webserver_port=44653
--webserver_interface=127.10.12.130
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:26:03.948870 27128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:03.945936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.129:46351)
0924 03:26:03.946955 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:03.947318 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":384,"threads_started":1}
W20260924 03:26:03.949750 27126 heartbeater.cc:644] Failed to heartbeat to 127.10.12.190:39019 (0 consecutive failures): Network error: Failed to ping master at 127.10.12.190:39019: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
I20260924 03:26:03.950965 27128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:03.950458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.129:44841)
0924 03:26:03.950680 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:03.950789 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 03:26:03.952283 27128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:03.951751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.129:44375)
0924 03:26:03.951922 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:03.952084 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 03:26:03.952615 27126 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:26:04.399730 27129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:04.400175 27129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:04.400345 27129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:04.409075 27129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:04.409447 27129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:04.409698 27129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:26:04.423384 27129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:44323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=44653
--tserver_master_addrs=127.10.12.190:39019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:04.425323 27129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:04.427160 27129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:04.445596 27135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:04.445549 27137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:04.445549 27134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:04.450629 27129 server_base.cc:1034] running on GCE node
I20260924 03:26:04.451704 27129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:04.453452 27129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:04.454777 27129 hybrid_clock.cc:648] HybridClock initialized: now 1790220364454670 us; error 132 us; skew 500 ppm
I20260924 03:26:04.455261 27129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:26:04 dist-test-slave-00qs krb5kdc[25798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220364, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:26:04.459810 27129 init.cc:377] Logged in from keytab as kudu/127.10.12.130@KRBTEST.COM (short username kudu)
I20260924 03:26:04.462340 27129 webserver.cc:460] Webserver started at http://127.10.12.130:44653/ using document root <none> and password file <none>
I20260924 03:26:04.463706 27129 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:04.463899 27129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:04.472297 27129 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 03:26:04.476662 27144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:04.479074 27129 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260924 03:26:04.479369 27129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "a5d08a1e9e764220811433108eb7dcbe"
format_stamp: "Formatted at 2026-09-24 03:25:16 on dist-test-slave-00qs"
server_key: "329f2709cd00c5ba8317072c59970e7d"
server_key_iv: "fa5778828f770eebe6b1105185f46267"
server_key_version: "encryptionkey@0"
I20260924 03:26:04.480880 27129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:04.498171 27129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:04.503760 27129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:04.504161 27129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:04.505416 27129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:26:04.507210 27129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:26:04.507330 27129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:04.507467 27129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:26:04.507543 27129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:04.548410 27129 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:44323
I20260924 03:26:04.548543 27256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:44323 every 8 connection(s)
I20260924 03:26:04.550731 27129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:26:04.553200 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 27129
I20260924 03:26:04.554062 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:40007
--local_ip_for_outbound_sockets=127.10.12.131
--tserver_master_addrs=127.10.12.190:39019
--webserver_port=35227
--webserver_interface=127.10.12.131
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:26:04.565685 27259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:04.555842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.130:55393)
0924 03:26:04.563818 (+ 7976us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:04.564279 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7504,"thread_start_us":7146,"threads_started":1}
W20260924 03:26:04.566839 27257 heartbeater.cc:644] Failed to heartbeat to 127.10.12.190:39019 (0 consecutive failures): Network error: Failed to ping master at 127.10.12.190:39019: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
I20260924 03:26:04.568857 27259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:04.567875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.130:56089)
0924 03:26:04.568302 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:04.568435 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254}
I20260924 03:26:04.570519 27259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:04.569980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.130:43287)
0924 03:26:04.570185 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:04.570319 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260924 03:26:04.570907 27257 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 03:26:04.954766 27261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:04.953599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.129:35617)
0924 03:26:04.954327 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:04.954455 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":359,"threads_started":1}
W20260924 03:26:04.985862 27260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:04.986282 27260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:04.986469 27260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:04.995909 27260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:04.996186 27260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:04.996338 27260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:26:05.015323 27260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:40007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=35227
--tserver_master_addrs=127.10.12.190:39019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:05.017378 27260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:05.019276 27260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:05.042145 27266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:05.042061 27269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:05.042061 27267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:05.043604 27260 server_base.cc:1034] running on GCE node
I20260924 03:26:05.044600 27260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:05.046167 27260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:05.047642 27260 hybrid_clock.cc:648] HybridClock initialized: now 1790220365047566 us; error 66 us; skew 500 ppm
I20260924 03:26:05.048213 27260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:26:05 dist-test-slave-00qs krb5kdc[25798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220365, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:26:05.052522 27260 init.cc:377] Logged in from keytab as kudu/127.10.12.131@KRBTEST.COM (short username kudu)
I20260924 03:26:05.054773 27260 webserver.cc:460] Webserver started at http://127.10.12.131:35227/ using document root <none> and password file <none>
I20260924 03:26:05.055721 27260 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:05.055879 27260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:05.064100 27260 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20260924 03:26:05.068368 27276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:05.070518 27260 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260924 03:26:05.070777 27260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "cd9dfeb5e9404919b0c4d22a61a6518a"
format_stamp: "Formatted at 2026-09-24 03:25:17 on dist-test-slave-00qs"
server_key: "bba68c1301e128a321885e4ecdcae5fd"
server_key_iv: "a2797c99994ccd33becb02efd70ac660"
server_key_version: "encryptionkey@0"
I20260924 03:26:05.072211 27260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:05.099943 27260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:05.106220 27260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:05.106570 27260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:05.108003 27260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:26:05.109907 27260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:26:05.110009 27260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:05.110133 27260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:26:05.110200 27260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:05.150313 27260 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:40007
I20260924 03:26:05.150370 27388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:40007 every 8 connection(s)
I20260924 03:26:05.152271 27260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:26:05.154182 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 27260
I20260924 03:26:05.160115 27392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:05.157144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.131:42983)
0924 03:26:05.158303 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:05.158702 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":766,"thread_start_us":491,"threads_started":1}
W20260924 03:26:05.161001 27389 heartbeater.cc:644] Failed to heartbeat to 127.10.12.190:39019 (0 consecutive failures): Network error: Failed to ping master at 127.10.12.190:39019: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
I20260924 03:26:05.162250 27392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:05.161782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.131:40943)
0924 03:26:05.161989 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:05.162084 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 03:26:05.163514 27392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:05.163118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.131:36799)
0924 03:26:05.163275 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:05.163365 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 03:26:05.163892 27389 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 03:26:05.572804 27393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:05.571685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.130:40975)
0924 03:26:05.572321 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:05.572455 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":332,"threads_started":1}
I20260924 03:26:05.957108 27394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:05.956203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.129:59197)
0924 03:26:05.956776 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:05.956875 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":308,"threads_started":1}
I20260924 03:26:06.166145 27395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:06.165054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.131:50375)
0924 03:26:06.165719 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:06.165845 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":289,"threads_started":1}
I20260924 03:26:06.575037 27396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:06.574019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.130:51533)
0924 03:26:06.574672 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:06.574795 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":317,"threads_started":1}
I20260924 03:26:06.959553 27397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:06.958493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.129:49461)
0924 03:26:06.959153 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:06.959312 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":322,"threads_started":1}
I20260924 03:26:07.168506 27398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:07.167478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.131:56379)
0924 03:26:07.168146 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:07.168271 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":326,"threads_started":1}
I20260924 03:26:07.578068 27399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:07.576623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.130:52747)
0924 03:26:07.577593 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:07.577822 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":337,"threads_started":1}
I20260924 03:26:07.962013 27400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:07.960953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.129:35123)
0924 03:26:07.961627 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:07.961785 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":320,"threads_started":1}
I20260924 03:26:08.170644 27401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:08.169762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.131:36523)
0924 03:26:08.170318 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:08.170414 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":284,"threads_started":1}
I20260924 03:26:08.580509 27402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:08.579428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.130:51241)
0924 03:26:08.580128 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:08.580281 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":374,"threads_started":1}
I20260924 03:26:08.964388 27403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:08.963413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.129:42943)
0924 03:26:08.964025 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:08.964153 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":291,"threads_started":1}
I20260924 03:26:09.173038 27404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:09.172021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.131:53035)
0924 03:26:09.172713 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:09.172815 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":323,"threads_started":1}
I20260924 03:26:09.582772 27405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:09.581758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.130:51329)
0924 03:26:09.582423 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:09.582523 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":369,"threads_started":1}
I20260924 03:26:09.966722 27406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:09.965748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.129:59085)
0924 03:26:09.966394 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:09.966510 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":307,"threads_started":1}
I20260924 03:26:10.175329 27407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:10.174334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.131:39237)
0924 03:26:10.174974 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:10.175086 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":346,"threads_started":1}
I20260924 03:26:10.585235 27408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:10.584145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.130:45033)
0924 03:26:10.584824 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:10.584962 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":311,"threads_started":1}
I20260924 03:26:10.969157 27409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:10.968096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.129:59051)
0924 03:26:10.968767 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:10.968880 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":314,"threads_started":1}
I20260924 03:26:11.177824 27410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:11.176823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.131:59347)
0924 03:26:11.177466 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:11.177589 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":353,"threads_started":1}
I20260924 03:26:11.587512 27411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:11.586425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.130:55089)
0924 03:26:11.587116 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:11.587233 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":344,"threads_started":1}
I20260924 03:26:11.971554 27412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:11.970456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.129:59393)
0924 03:26:11.971157 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:11.971287 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":400,"threads_started":1}
I20260924 03:26:12.179774 27413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:12.178903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.131:43783)
0924 03:26:12.179464 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:12.179552 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":253,"threads_started":1}
I20260924 03:26:12.590458 27414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:12.588856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.130:45949)
0924 03:26:12.589763 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:12.589955 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":443,"threads_started":1}
I20260924 03:26:12.973693 27415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:12.972862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.129:58245)
0924 03:26:12.973390 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:12.973504 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":242,"threads_started":1}
I20260924 03:26:13.181638 27416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:13.180721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.131:57215)
0924 03:26:13.181292 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:13.181419 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":335,"threads_started":1}
I20260924 03:26:13.592419 27417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:13.591507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.130:50427)
0924 03:26:13.592134 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:13.592230 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":301,"threads_started":1}
I20260924 03:26:13.975806 27418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:13.974935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.129:50263)
0924 03:26:13.975469 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:13.975584 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":276,"threads_started":1}
I20260924 03:26:14.184021 27419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:14.182935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.131:36565)
0924 03:26:14.183594 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:14.183725 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":338,"threads_started":1}
I20260924 03:26:14.594950 27420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:14.593796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.130:39033)
0924 03:26:14.594498 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:14.594639 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":363,"threads_started":1}
I20260924 03:26:14.978144 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:14.977003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.129:41243)
0924 03:26:14.977736 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:14.977876 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":358,"threads_started":1}
I20260924 03:26:15.186520 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:15.185409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.131:35645)
0924 03:26:15.186126 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:15.186268 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":374,"threads_started":1}
I20260924 03:26:15.597286 27423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:15.596334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.130:44199)
0924 03:26:15.596918 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:15.597038 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":281,"threads_started":1}
I20260924 03:26:15.980703 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:15.979511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.129:56863)
0924 03:26:15.980259 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:15.980428 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":406,"threads_started":1}
I20260924 03:26:16.188680 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:16.187776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.131:57933)
0924 03:26:16.188323 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:16.188441 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":282,"threads_started":1}
I20260924 03:26:16.599404 27426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:16.598491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.130:44053)
0924 03:26:16.599096 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:16.599215 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":256,"threads_started":1}
I20260924 03:26:16.982580 27427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:16.981744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.129:43843)
0924 03:26:16.982252 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:16.982368 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":230,"threads_started":1}
I20260924 03:26:17.190512 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:17.189731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.131:58153)
0924 03:26:17.190199 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:17.190307 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":232,"threads_started":1}
I20260924 03:26:17.601258 27429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:17.600451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.130:52099)
0924 03:26:17.600963 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:17.601068 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":238,"threads_started":1}
I20260924 03:26:17.992818 27430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:17.983598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.129:47513)
0924 03:26:17.992508 (+ 8910us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:17.992621 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8764,"thread_start_us":8565,"threads_started":1}
I20260924 03:26:18.192557 27431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:18.191630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.131:38541)
0924 03:26:18.192199 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:18.192363 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":341,"threads_started":1}
I20260924 03:26:18.603109 27432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:18.602323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.130:41267)
0924 03:26:18.602798 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:18.602927 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":241,"threads_started":1}
I20260924 03:26:18.994848 27433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:18.994049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.129:49643)
0924 03:26:18.994564 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:18.994664 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":230,"threads_started":1}
I20260924 03:26:19.194271 27434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:19.193495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.131:57751)
0924 03:26:19.193997 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:19.194083 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":247,"threads_started":1}
I20260924 03:26:19.605603 27435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:19.604678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.130:58337)
0924 03:26:19.605236 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:19.605359 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":273,"threads_started":1}
I20260924 03:26:19.997179 27436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:19.996160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.129:58705)
0924 03:26:19.996842 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:19.996961 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":331,"threads_started":1}
I20260924 03:26:20.158865 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 26957
I20260924 03:26:20.188684 27437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:20.160543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58986 (local address 127.10.12.129:42927)
0924 03:26:20.161485 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:20.161508 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:20.161562 (+ 54us) server_negotiation.cc:407] Connection header received
0924 03:26:20.163653 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:20.163677 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:20.164387 (+ 710us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:20.164819 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:20.166812 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:20.168422 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:20.169995 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:20.170878 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:20.171105 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:20.179629 (+ 8524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:20.179643 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:20.179670 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:20.179742 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:20.183669 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:20.184840 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:20.184852 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:20.184863 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:20.185124 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:20.185694 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:20.185700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:20.185702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:20.186768 (+ 1066us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:20.187094 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:20.187698 (+ 604us) server_negotiation.cc:299] Negotiation successful
0924 03:26:20.188294 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":483,"threads_started":1}
I20260924 03:26:20.190647 27121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:20.196902 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:20.195780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:39019 (local address 127.10.12.131:57115)
0924 03:26:20.196413 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:20.196583 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.190:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":284,"threads_started":1}
I20260924 03:26:20.288231 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 26957
I20260924 03:26:20.304636 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 27129
I20260924 03:26:20.324273 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:20.305747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37824 (local address 127.10.12.130:44323)
0924 03:26:20.306307 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:20.306322 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:20.306366 (+ 44us) server_negotiation.cc:407] Connection header received
0924 03:26:20.308095 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:20.308108 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:20.308607 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:20.308944 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:20.310463 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:20.311879 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:20.313328 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:20.314195 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:20.314365 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:20.317357 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:20.317366 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:20.317386 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:20.317481 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:20.320686 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:20.321556 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:20.321564 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:20.321570 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:20.321768 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:20.322155 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:20.322158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:20.322160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:20.322798 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:20.323040 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:20.323447 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 03:26:20.323868 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":227,"threads_started":1}
I20260924 03:26:20.325855 27252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:20.412307 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 27129
I20260924 03:26:20.427089 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 27260
I20260924 03:26:20.447209 27442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:20.428343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35178 (local address 127.10.12.131:40007)
0924 03:26:20.429110 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:20.429125 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:20.429170 (+ 45us) server_negotiation.cc:407] Connection header received
0924 03:26:20.430919 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:20.430930 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:20.431401 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:20.431728 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:20.433245 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:20.434561 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:20.436038 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:20.436747 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:20.436909 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:20.439719 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:20.439730 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:20.439748 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:20.439800 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:20.443170 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:20.444258 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:20.444267 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:20.444274 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:20.444474 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:20.444988 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:20.444993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:20.444995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:20.445627 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:20.445894 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:20.446401 (+ 507us) server_negotiation.cc:299] Negotiation successful
0924 03:26:20.446860 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":338,"threads_started":1}
I20260924 03:26:20.448820 27384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:20.534842 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 27260
2026-09-24T03:26:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (66818 ms)
[----------] 30 tests from SecurityITest (313224 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:26:20 dist-test-slave-00qs krb5kdc[27446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:26:20 dist-test-slave-00qs krb5kdc[27446](info): set up 2 sockets
Sep 24 03:26:20 dist-test-slave-00qs krb5kdc[27446](info): commencing operation
krb5kdc: starting...
W20260924 03:26:21.568701 10290 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:26:21 dist-test-slave-00qs krb5kdc[27446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:26:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:26:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:26:21.820999 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:33141
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:33141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:26:22.185426 27459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:22.185742 27459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:22.185863 27459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:22.193333 27459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:26:22.193463 27459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:22.193509 27459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:22.193575 27459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:26:22.193636 27459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:26:22.203676 27459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:33141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:33141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:22.205096 27459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:22.206431 27459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:22.218916 27464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:22.218883 27465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:22.218910 27467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:22.219283 27459 server_base.cc:1034] running on GCE node
I20260924 03:26:22.220486 27459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:22.222067 27459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:22.223295 27459 hybrid_clock.cc:648] HybridClock initialized: now 1790220382223240 us; error 41 us; skew 500 ppm
I20260924 03:26:22.223664 27459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:26:22 dist-test-slave-00qs krb5kdc[27446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220382, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:26:22.227015 27459 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:26:22.228873 27459 webserver.cc:460] Webserver started at http://127.10.12.190:43267/ using document root <none> and password file <none>
I20260924 03:26:22.229636 27459 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:22.229765 27459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:22.230088 27459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:22.233460 27459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "9e308e5ef9e144d5ba0d88c7d170333d"
format_stamp: "Formatted at 2026-09-24 03:26:22 on dist-test-slave-00qs"
server_key: "206ac92a3d3287919d25a6987416f8aa"
server_key_iv: "263e6bd0315ac81c0174e5ee3eae209a"
server_key_version: "encryptionkey@0"
I20260924 03:26:22.234258 27459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "9e308e5ef9e144d5ba0d88c7d170333d"
format_stamp: "Formatted at 2026-09-24 03:26:22 on dist-test-slave-00qs"
server_key: "206ac92a3d3287919d25a6987416f8aa"
server_key_iv: "263e6bd0315ac81c0174e5ee3eae209a"
server_key_version: "encryptionkey@0"
I20260924 03:26:22.239805 27459 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 03:26:22.243552 27474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:22.245128 27459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:26:22.245402 27459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "9e308e5ef9e144d5ba0d88c7d170333d"
format_stamp: "Formatted at 2026-09-24 03:26:22 on dist-test-slave-00qs"
server_key: "206ac92a3d3287919d25a6987416f8aa"
server_key_iv: "263e6bd0315ac81c0174e5ee3eae209a"
server_key_version: "encryptionkey@0"
I20260924 03:26:22.245620 27459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:22.264113 27459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:22.268704 27459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:22.269003 27459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:22.287940 27459 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:33141
I20260924 03:26:22.287986 27525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:33141 every 8 connection(s)
I20260924 03:26:22.289414 27459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:26:22.293485 27526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:22.296365 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 27459
I20260924 03:26:22.296574 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:26:22.297396 10290 external_mini_cluster.cc:1428] Setting key 0a40e3001718adbbb70f8cb25e3cd280
I20260924 03:26:22.305918 27526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e308e5ef9e144d5ba0d88c7d170333d: Bootstrap starting.
I20260924 03:26:22.309127 27526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e308e5ef9e144d5ba0d88c7d170333d: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:22.310233 27526 log.cc:824] T 00000000000000000000000000000000 P 9e308e5ef9e144d5ba0d88c7d170333d: Log is configured to *not* fsync() on all Append() calls
Sep 24 03:26:22 dist-test-slave-00qs krb5kdc[27446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:26:22.313889 27526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e308e5ef9e144d5ba0d88c7d170333d: No bootstrap required, opened a new log
I20260924 03:26:22.321542 27529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:22.299680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51092 (local address 127.10.12.190:33141)
0924 03:26:22.300586 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:22.300603 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:22.300647 (+ 44us) server_negotiation.cc:407] Connection header received
0924 03:26:22.302405 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:22.302436 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:22.302873 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:22.303257 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:22.304802 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:22.306144 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:22.307585 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:22.308322 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:22.308534 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:22.313456 (+ 4922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:22.313465 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:22.313482 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:22.313523 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:22.316848 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:22.317847 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:22.317856 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:22.317863 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:22.318063 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:22.318493 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:22.318497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:22.318499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:22.319106 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:22.319349 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:22.319841 (+ 492us) server_negotiation.cc:299] Negotiation successful
0924 03:26:22.320320 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":391,"threads_started":1}
I20260924 03:26:22.322961 27526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e308e5ef9e144d5ba0d88c7d170333d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e308e5ef9e144d5ba0d88c7d170333d" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 33141 } }
I20260924 03:26:22.323446 27526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e308e5ef9e144d5ba0d88c7d170333d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:22.323555 27526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e308e5ef9e144d5ba0d88c7d170333d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e308e5ef9e144d5ba0d88c7d170333d, State: Initialized, Role: FOLLOWER
I20260924 03:26:22.324231 27526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e308e5ef9e144d5ba0d88c7d170333d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e308e5ef9e144d5ba0d88c7d170333d" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 33141 } }
I20260924 03:26:22.324538 27526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e308e5ef9e144d5ba0d88c7d170333d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:26:22.324713 27526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e308e5ef9e144d5ba0d88c7d170333d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:26:22.324867 27526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e308e5ef9e144d5ba0d88c7d170333d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:22.327040 27526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e308e5ef9e144d5ba0d88c7d170333d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e308e5ef9e144d5ba0d88c7d170333d" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 33141 } }
I20260924 03:26:22.327566 27526 leader_election.cc:302] T 00000000000000000000000000000000 P 9e308e5ef9e144d5ba0d88c7d170333d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e308e5ef9e144d5ba0d88c7d170333d; no voters:
I20260924 03:26:22.328190 27526 leader_election.cc:288] T 00000000000000000000000000000000 P 9e308e5ef9e144d5ba0d88c7d170333d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:26:22.328317 27531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e308e5ef9e144d5ba0d88c7d170333d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:26:22.328963 27531 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e308e5ef9e144d5ba0d88c7d170333d [term 1 LEADER]: Becoming Leader. State: Replica: 9e308e5ef9e144d5ba0d88c7d170333d, State: Running, Role: LEADER
I20260924 03:26:22.329478 27531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e308e5ef9e144d5ba0d88c7d170333d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e308e5ef9e144d5ba0d88c7d170333d" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 33141 } }
I20260924 03:26:22.330242 27526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e308e5ef9e144d5ba0d88c7d170333d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:26:22.332391 27532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e308e5ef9e144d5ba0d88c7d170333d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e308e5ef9e144d5ba0d88c7d170333d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e308e5ef9e144d5ba0d88c7d170333d" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 33141 } } }
I20260924 03:26:22.332849 27532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e308e5ef9e144d5ba0d88c7d170333d [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:22.333702 27538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:26:22.333566 27533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e308e5ef9e144d5ba0d88c7d170333d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e308e5ef9e144d5ba0d88c7d170333d. Latest consensus state: current_term: 1 leader_uuid: "9e308e5ef9e144d5ba0d88c7d170333d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e308e5ef9e144d5ba0d88c7d170333d" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 33141 } } }
I20260924 03:26:22.333992 27533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e308e5ef9e144d5ba0d88c7d170333d [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:22.338951 27538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:26:22.348450 27538 catalog_manager.cc:1347] Generated new cluster ID: 1ecb6c5335864f779564fa1561c85296
I20260924 03:26:22.348577 27538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:26:22.366619 27538 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:26:22.368155 27538 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:26:22.377478 27538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e308e5ef9e144d5ba0d88c7d170333d: Generated new TSK 0
I20260924 03:26:22.378468 27538 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
I20260924 03:26:22.457044 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:33141
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:26:22.923179 27552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:22.923487 27552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:22.923616 27552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:22.931146 27552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:22.931283 27552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:22.931438 27552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:26:22.942817 27552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:33141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:22.944280 27552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:22.945616 27552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:22.960019 27557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:22.959920 27560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:22.959987 27558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:22.965958 27552 server_base.cc:1034] running on GCE node
I20260924 03:26:22.966683 27552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:22.967732 27552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:22.968969 27552 hybrid_clock.cc:648] HybridClock initialized: now 1790220382968890 us; error 67 us; skew 500 ppm
I20260924 03:26:22.969337 27552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:26:22 dist-test-slave-00qs krb5kdc[27446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220382, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:26:22.972779 27552 init.cc:377] Logged in from keytab as kudu/127.10.12.129@KRBTEST.COM (short username kudu)
I20260924 03:26:22.974488 27552 webserver.cc:460] Webserver started at http://127.10.12.129:39231/ using document root <none> and password file <none>
I20260924 03:26:22.975245 27552 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:22.975359 27552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:22.975678 27552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:22.978876 27552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "9f1afa34590b4717a936f23cb95ba36a"
format_stamp: "Formatted at 2026-09-24 03:26:22 on dist-test-slave-00qs"
server_key: "060a89ad09e57849fefc989681eed65b"
server_key_iv: "16d90eb010bbf8678db5176c27529f94"
server_key_version: "encryptionkey@0"
I20260924 03:26:22.979736 27552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "9f1afa34590b4717a936f23cb95ba36a"
format_stamp: "Formatted at 2026-09-24 03:26:22 on dist-test-slave-00qs"
server_key: "060a89ad09e57849fefc989681eed65b"
server_key_iv: "16d90eb010bbf8678db5176c27529f94"
server_key_version: "encryptionkey@0"
I20260924 03:26:22.985330 27552 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260924 03:26:22.989382 27567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:22.990883 27552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:26:22.991068 27552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "9f1afa34590b4717a936f23cb95ba36a"
format_stamp: "Formatted at 2026-09-24 03:26:22 on dist-test-slave-00qs"
server_key: "060a89ad09e57849fefc989681eed65b"
server_key_iv: "16d90eb010bbf8678db5176c27529f94"
server_key_version: "encryptionkey@0"
I20260924 03:26:22.991276 27552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:23.010699 27552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:23.015338 27552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:23.015632 27552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:23.016695 27552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:26:23.018200 27552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:26:23.018285 27552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:23.018411 27552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:26:23.018471 27552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:23.051795 27552 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:43375
I20260924 03:26:23.051841 27679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:43375 every 8 connection(s)
I20260924 03:26:23.053133 27552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:26:23.061290 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 27552
I20260924 03:26:23.061510 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:26:23.062173 10290 external_mini_cluster.cc:1428] Setting key 2c20a38723cf5263d4d6b2bcabc4fc71
Sep 24 03:26:23 dist-test-slave-00qs krb5kdc[27446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220382, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:26:23.074637 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:23.055796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:55359 (local address 127.10.12.190:33141)
0924 03:26:23.056279 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:23.056284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:23.057159 (+ 875us) server_negotiation.cc:407] Connection header received
0924 03:26:23.059119 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:23.059126 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:23.059213 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:23.059309 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:23.061006 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:23.061817 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:23.063600 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:23.064128 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:23.064241 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:23.068408 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:23.068414 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:23.068417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:23.068456 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:23.070965 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:23.071783 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:23.071788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:23.071790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:23.072008 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:23.072629 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:23.072635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:23.072638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:23.073035 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:23.073172 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:23.074025 (+ 853us) server_negotiation.cc:299] Negotiation successful
0924 03:26:23.074269 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":257,"threads_started":1}
I20260924 03:26:23.075441 27683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:23.056284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:33141 (local address 127.10.12.129:55359)
0924 03:26:23.057005 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:23.057042 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:23.058815 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:23.059576 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:23.059586 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:23.060220 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:26:23.060816 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:23.060834 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:23.061982 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:23.061986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:23.063458 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:23.063472 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:23.064360 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:23.064367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:23.064622 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:23.065695 (+ 1073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:26:23.065720 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:26:23.068124 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:26:23.071142 (+ 3018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:26:23.071152 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:26:23.071168 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:26:23.071638 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:26:23.072157 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:26:23.072161 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:26:23.072163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:26:23.072441 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:26:23.073183 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:26:23.073190 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:26:23.073611 (+ 421us) client_negotiation.cc:769] Sending connection context
0924 03:26:23.073877 (+ 266us) client_negotiation.cc:240] Negotiation successful
0924 03:26:23.074304 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":278,"threads_started":1}
I20260924 03:26:23.077203 27680 heartbeater.cc:347] Connected to a master server at 127.10.12.190:33141
I20260924 03:26:23.077488 27680 heartbeater.cc:464] Registering TS with master...
I20260924 03:26:23.078263 27680 heartbeater.cc:511] Master 127.10.12.190:33141 requested a full tablet report, sending...
I20260924 03:26:23.080590 27491 ts_manager.cc:194] Registered new tserver with Master: 9f1afa34590b4717a936f23cb95ba36a (127.10.12.129:43375)
I20260924 03:26:23.082247 27491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.129@KRBTEST.COM'} at 127.10.12.129:55359
WARNING: no policy specified for kudu/127.10.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
I20260924 03:26:23.121181 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:33141
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:26:23.480069 27686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:23.480459 27686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:23.480615 27686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:23.488233 27686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:23.488401 27686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:23.488576 27686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:26:23.498910 27686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:33141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:23.500334 27686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:23.501811 27686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:23.514966 27691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:23.515008 27692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:23.515172 27694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:23.516120 27686 server_base.cc:1034] running on GCE node
I20260924 03:26:23.516758 27686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:23.517716 27686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:23.518960 27686 hybrid_clock.cc:648] HybridClock initialized: now 1790220383518865 us; error 82 us; skew 500 ppm
I20260924 03:26:23.519287 27686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:26:23 dist-test-slave-00qs krb5kdc[27446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220383, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:26:23.522534 27686 init.cc:377] Logged in from keytab as kudu/127.10.12.130@KRBTEST.COM (short username kudu)
I20260924 03:26:23.523994 27686 webserver.cc:460] Webserver started at http://127.10.12.130:46407/ using document root <none> and password file <none>
I20260924 03:26:23.524703 27686 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:23.524811 27686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:23.525076 27686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:23.528055 27686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "6d0793a729ba4ce799e741f50c6d532d"
format_stamp: "Formatted at 2026-09-24 03:26:23 on dist-test-slave-00qs"
server_key: "efe1d58ff5a90ba20991436bd2e1e739"
server_key_iv: "fc2d67a09a9d310f79a4c3f05fb83dee"
server_key_version: "encryptionkey@0"
I20260924 03:26:23.528810 27686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "6d0793a729ba4ce799e741f50c6d532d"
format_stamp: "Formatted at 2026-09-24 03:26:23 on dist-test-slave-00qs"
server_key: "efe1d58ff5a90ba20991436bd2e1e739"
server_key_iv: "fc2d67a09a9d310f79a4c3f05fb83dee"
server_key_version: "encryptionkey@0"
I20260924 03:26:23.534205 27686 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 03:26:23.537918 27701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:23.539314 27686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:26:23.539491 27686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "6d0793a729ba4ce799e741f50c6d532d"
format_stamp: "Formatted at 2026-09-24 03:26:23 on dist-test-slave-00qs"
server_key: "efe1d58ff5a90ba20991436bd2e1e739"
server_key_iv: "fc2d67a09a9d310f79a4c3f05fb83dee"
server_key_version: "encryptionkey@0"
I20260924 03:26:23.539726 27686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:23.555611 27686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:23.560091 27686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:23.560387 27686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:23.561424 27686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:26:23.562837 27686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:26:23.562915 27686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:23.563014 27686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:26:23.563071 27686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:23.595541 27686 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:45013
I20260924 03:26:23.595598 27813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:45013 every 8 connection(s)
I20260924 03:26:23.596987 27686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:26:23.607373 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 27686
I20260924 03:26:23.607544 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:26:23.608067 10290 external_mini_cluster.cc:1428] Setting key c5cbffa5df83218823bb6941f8cbcd13
Sep 24 03:26:23 dist-test-slave-00qs krb5kdc[27446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220383, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:26:23.618330 27816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:23.599669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:40605 (local address 127.10.12.190:33141)
0924 03:26:23.600195 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:23.600200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:23.601146 (+ 946us) server_negotiation.cc:407] Connection header received
0924 03:26:23.603041 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:23.603047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:23.603133 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:23.603231 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:23.604919 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:23.605641 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:23.607388 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:23.607911 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:23.608069 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:23.612390 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:23.612396 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:23.612400 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:23.612437 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:23.615050 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:23.615888 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:23.615892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:23.615895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:23.616068 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:23.616512 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:23.616517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:23.616519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:23.616809 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:23.616956 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:23.617702 (+ 746us) server_negotiation.cc:299] Negotiation successful
0924 03:26:23.617954 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":307,"threads_started":1}
I20260924 03:26:23.619308 27817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:23.600156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:33141 (local address 127.10.12.130:40605)
0924 03:26:23.600956 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:23.601008 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:23.602730 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:23.603496 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:23.603505 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:23.604140 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:26:23.604736 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:23.604754 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:23.605793 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:23.605799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:23.607154 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:23.607172 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:23.608188 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:23.608196 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:23.608427 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:23.609357 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:26:23.609390 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:26:23.612133 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:26:23.615242 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:26:23.615251 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:26:23.615267 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:26:23.615741 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:26:23.616176 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:26:23.616181 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:26:23.616185 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:26:23.616393 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:26:23.616938 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:26:23.616947 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:26:23.617314 (+ 367us) client_negotiation.cc:769] Sending connection context
0924 03:26:23.617572 (+ 258us) client_negotiation.cc:240] Negotiation successful
0924 03:26:23.618130 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":347,"threads_started":1}
I20260924 03:26:23.620819 27814 heartbeater.cc:347] Connected to a master server at 127.10.12.190:33141
I20260924 03:26:23.621145 27814 heartbeater.cc:464] Registering TS with master...
I20260924 03:26:23.621860 27814 heartbeater.cc:511] Master 127.10.12.190:33141 requested a full tablet report, sending...
I20260924 03:26:23.623415 27491 ts_manager.cc:194] Registered new tserver with Master: 6d0793a729ba4ce799e741f50c6d532d (127.10.12.130:45013)
I20260924 03:26:23.624507 27491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.130@KRBTEST.COM'} at 127.10.12.130:40605
WARNING: no policy specified for kudu/127.10.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
I20260924 03:26:23.665033 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:33141
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:26:24.000680 27820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:24.001005 27820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:24.001144 27820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:24.009043 27820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:24.009188 27820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:24.009342 27820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:26:24.019500 27820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:33141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:24.020870 27820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:24.022218 27820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:24.035934 27825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:24.035876 27828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:24.036068 27826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:24.036805 27820 server_base.cc:1034] running on GCE node
I20260924 03:26:24.037530 27820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:24.038563 27820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:24.039770 27820 hybrid_clock.cc:648] HybridClock initialized: now 1790220384039724 us; error 42 us; skew 500 ppm
I20260924 03:26:24.040133 27820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:26:24 dist-test-slave-00qs krb5kdc[27446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220384, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:26:24.043251 27820 init.cc:377] Logged in from keytab as kudu/127.10.12.131@KRBTEST.COM (short username kudu)
I20260924 03:26:24.044938 27820 webserver.cc:460] Webserver started at http://127.10.12.131:46377/ using document root <none> and password file <none>
I20260924 03:26:24.045641 27820 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:24.045742 27820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:24.046015 27820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:24.049052 27820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "4e6fcee514da46cc9c23ce84f3fa4925"
format_stamp: "Formatted at 2026-09-24 03:26:24 on dist-test-slave-00qs"
server_key: "6a4a3f239db48a69582a5fc02753bdb8"
server_key_iv: "aaf2a92cc0487a3eda92e47d7bf8cd54"
server_key_version: "encryptionkey@0"
I20260924 03:26:24.049831 27820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "4e6fcee514da46cc9c23ce84f3fa4925"
format_stamp: "Formatted at 2026-09-24 03:26:24 on dist-test-slave-00qs"
server_key: "6a4a3f239db48a69582a5fc02753bdb8"
server_key_iv: "aaf2a92cc0487a3eda92e47d7bf8cd54"
server_key_version: "encryptionkey@0"
I20260924 03:26:24.055280 27820 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:26:24.059178 27835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:24.060815 27820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:26:24.061002 27820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "4e6fcee514da46cc9c23ce84f3fa4925"
format_stamp: "Formatted at 2026-09-24 03:26:24 on dist-test-slave-00qs"
server_key: "6a4a3f239db48a69582a5fc02753bdb8"
server_key_iv: "aaf2a92cc0487a3eda92e47d7bf8cd54"
server_key_version: "encryptionkey@0"
I20260924 03:26:24.061219 27820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:24.085618 27680 heartbeater.cc:503] Master 127.10.12.190:33141 was elected leader, sending a full tablet report...
I20260924 03:26:24.087644 27820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:24.092659 27820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:24.092999 27820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:24.094043 27820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:26:24.095592 27820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:26:24.095672 27820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:24.095813 27820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:26:24.095870 27820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:24.134986 27820 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:39609
I20260924 03:26:24.135025 27947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:39609 every 8 connection(s)
I20260924 03:26:24.136318 27820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:26:24.144819 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 27820
I20260924 03:26:24.144991 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:26:24.145435 10290 external_mini_cluster.cc:1428] Setting key 40601509b79ea043720075ea0d799792
Sep 24 03:26:24 dist-test-slave-00qs krb5kdc[27446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220384, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:26:24.158229 27950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:24.138938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:54155 (local address 127.10.12.190:33141)
0924 03:26:24.139456 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:24.139463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:24.140641 (+ 1178us) server_negotiation.cc:407] Connection header received
0924 03:26:24.142603 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:24.142609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:24.142695 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:24.142798 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:24.144723 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.145689 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.147624 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.148082 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.148198 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:24.152214 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:24.152220 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:24.152224 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:24.152266 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:24.154904 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:24.155800 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:24.155805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:24.155807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:24.155985 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:24.156438 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:24.156442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:24.156444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:24.156733 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:24.156869 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:24.157608 (+ 739us) server_negotiation.cc:299] Negotiation successful
0924 03:26:24.157845 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":248,"threads_started":1}
I20260924 03:26:24.159308 27951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:24.139585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:33141 (local address 127.10.12.131:54155)
0924 03:26:24.140464 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:24.140513 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:24.142227 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:24.143090 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:24.143099 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:24.143746 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:26:24.144468 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:24.144496 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.145866 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.145871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:24.147472 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:24.147486 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.148310 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.148318 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:24.148550 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:24.149256 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:26:24.149281 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:26:24.151884 (+ 2603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:26:24.155124 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:26:24.155132 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:26:24.155147 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:26:24.155638 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:26:24.156109 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:26:24.156112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:26:24.156115 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:26:24.156313 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:26:24.156876 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:26:24.156886 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:26:24.157236 (+ 350us) client_negotiation.cc:769] Sending connection context
0924 03:26:24.157484 (+ 248us) client_negotiation.cc:240] Negotiation successful
0924 03:26:24.158058 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":314,"threads_started":1}
I20260924 03:26:24.160770 27948 heartbeater.cc:347] Connected to a master server at 127.10.12.190:33141
I20260924 03:26:24.161098 27948 heartbeater.cc:464] Registering TS with master...
I20260924 03:26:24.161800 27948 heartbeater.cc:511] Master 127.10.12.190:33141 requested a full tablet report, sending...
I20260924 03:26:24.163298 27491 ts_manager.cc:194] Registered new tserver with Master: 4e6fcee514da46cc9c23ce84f3fa4925 (127.10.12.131:39609)
I20260924 03:26:24.164166 27491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.131@KRBTEST.COM'} at 127.10.12.131:54155
I20260924 03:26:24.173663 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:26:24.193562 27950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:24.178495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51102 (local address 127.10.12.190:33141)
0924 03:26:24.178697 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:24.178702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:24.179081 (+ 379us) server_negotiation.cc:407] Connection header received
0924 03:26:24.179194 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:24.179198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:24.179284 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:24.179377 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:24.180908 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.181689 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.183538 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.184272 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.184410 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:24.187397 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:24.187403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:24.187407 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:24.187453 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:24.190257 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:24.191176 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:24.191180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:24.191183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:24.191328 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:24.191775 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:24.191779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:24.191782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:24.192218 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:24.192631 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:24.193006 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 03:26:24.193212 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 03:26:24.198274 27491 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:26:24.205327 27491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:26:24.247726 27965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:24.219847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59332 (local address 127.10.12.130:45013)
0924 03:26:24.238604 (+ 18757us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:24.238616 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:24.238654 (+ 38us) server_negotiation.cc:407] Connection header received
0924 03:26:24.238778 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:24.238787 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:24.239116 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:24.239341 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:24.240535 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.242170 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.244988 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.246464 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.246612 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:24.246827 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:24.247177 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 03:26:24.247373 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18540,"thread_start_us":20122,"threads_started":1}
I20260924 03:26:24.247768 27964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:24.219420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:45013 (local address 127.0.0.1:59332)
0924 03:26:24.236907 (+ 17487us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:24.236926 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:24.237082 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:24.239644 (+ 2562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:24.239652 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:24.239688 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:24.240322 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:24.240339 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.242326 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.242331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:24.244773 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:24.244788 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.246671 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.246677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:24.246888 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:24.246945 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 03:26:24.247084 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 03:26:24.247294 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17337,"thread_start_us":21810,"threads_started":1}
I20260924 03:26:24.248704 27962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:24.219734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:39609 (local address 127.0.0.1:60982)
0924 03:26:24.240338 (+ 20604us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:24.240357 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:24.240479 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:24.241433 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:24.241438 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:24.241463 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:24.241728 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:24.241735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.243492 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.243502 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:24.245870 (+ 2368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:24.245882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.247782 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.247790 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:24.248076 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:24.248108 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 03:26:24.248186 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 03:26:24.248321 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20446,"mutex_wait_us":56,"thread_start_us":16835,"threads_started":1}
I20260924 03:26:24.248878 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:24.220263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60982 (local address 127.10.12.131:39609)
0924 03:26:24.239227 (+ 18964us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:24.239234 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:24.240615 (+ 1381us) server_negotiation.cc:407] Connection header received
0924 03:26:24.240717 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:24.240724 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:24.240957 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:24.241140 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:24.241896 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.243339 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.246029 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.247251 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.247370 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:24.247592 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:24.248365 (+ 773us) server_negotiation.cc:299] Negotiation successful
0924 03:26:24.248556 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18794,"thread_start_us":18979,"threads_started":1}
I20260924 03:26:24.249512 27961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:24.219259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:43375 (local address 127.0.0.1:43782)
0924 03:26:24.233071 (+ 13812us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:24.233132 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:24.233400 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:24.240545 (+ 7145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:24.240549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:24.240581 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:24.240968 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:24.240981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.242976 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.242982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:24.245415 (+ 2433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:24.245431 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.247565 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.247572 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:24.247792 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:24.247885 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 03:26:24.248012 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 03:26:24.248150 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13454,"mutex_wait_us":30,"thread_start_us":13230,"threads_started":1}
I20260924 03:26:24.250393 27963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:24.219385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43782 (local address 127.10.12.129:43375)
0924 03:26:24.239938 (+ 20553us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:24.239946 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:24.240002 (+ 56us) server_negotiation.cc:407] Connection header received
0924 03:26:24.240094 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:24.240101 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:24.240328 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:24.241169 (+ 841us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:24.241553 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.242764 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.245604 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.247044 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.247180 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:24.247421 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:24.249842 (+ 2421us) server_negotiation.cc:299] Negotiation successful
0924 03:26:24.250040 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20393,"thread_start_us":20230,"threads_started":1}
I20260924 03:26:24.252363 27749 tablet_service.cc:1467] Processing CreateTablet for tablet 42b1dee0f9bf4b94bafd172a4437a2c5 (DEFAULT_TABLE table=test-table [id=3c16b840075a4812aa77d56c094dbdbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:26:24.252621 27883 tablet_service.cc:1467] Processing CreateTablet for tablet 42b1dee0f9bf4b94bafd172a4437a2c5 (DEFAULT_TABLE table=test-table [id=3c16b840075a4812aa77d56c094dbdbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:26:24.253935 27749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42b1dee0f9bf4b94bafd172a4437a2c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:24.254092 27883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42b1dee0f9bf4b94bafd172a4437a2c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:24.253998 27615 tablet_service.cc:1467] Processing CreateTablet for tablet 42b1dee0f9bf4b94bafd172a4437a2c5 (DEFAULT_TABLE table=test-table [id=3c16b840075a4812aa77d56c094dbdbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:26:24.255452 27615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42b1dee0f9bf4b94bafd172a4437a2c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:24.268839 27968 tablet_bootstrap.cc:492] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 6d0793a729ba4ce799e741f50c6d532d: Bootstrap starting.
I20260924 03:26:24.269366 27967 tablet_bootstrap.cc:492] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 4e6fcee514da46cc9c23ce84f3fa4925: Bootstrap starting.
I20260924 03:26:24.272828 27968 tablet_bootstrap.cc:654] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 6d0793a729ba4ce799e741f50c6d532d: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:24.273478 27969 tablet_bootstrap.cc:492] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a: Bootstrap starting.
I20260924 03:26:24.273618 27967 tablet_bootstrap.cc:654] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 4e6fcee514da46cc9c23ce84f3fa4925: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:24.274787 27968 log.cc:824] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 6d0793a729ba4ce799e741f50c6d532d: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:24.274858 27967 log.cc:824] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 4e6fcee514da46cc9c23ce84f3fa4925: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:24.276937 27969 tablet_bootstrap.cc:654] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:24.278257 27969 log.cc:824] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:24.281476 27969 tablet_bootstrap.cc:492] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a: No bootstrap required, opened a new log
I20260924 03:26:24.281742 27968 tablet_bootstrap.cc:492] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 6d0793a729ba4ce799e741f50c6d532d: No bootstrap required, opened a new log
I20260924 03:26:24.281792 27967 tablet_bootstrap.cc:492] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 4e6fcee514da46cc9c23ce84f3fa4925: No bootstrap required, opened a new log
I20260924 03:26:24.281776 27969 ts_tablet_manager.cc:1397] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260924 03:26:24.282042 27967 ts_tablet_manager.cc:1397] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 4e6fcee514da46cc9c23ce84f3fa4925: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20260924 03:26:24.282042 27968 ts_tablet_manager.cc:1397] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 6d0793a729ba4ce799e741f50c6d532d: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.010s
I20260924 03:26:24.290797 27969 raft_consensus.cc:348] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1afa34590b4717a936f23cb95ba36a" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 43375 } } peers { permanent_uuid: "6d0793a729ba4ce799e741f50c6d532d" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45013 } } peers { permanent_uuid: "4e6fcee514da46cc9c23ce84f3fa4925" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39609 } }
I20260924 03:26:24.291399 27969 raft_consensus.cc:374] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:24.291540 27969 raft_consensus.cc:729] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f1afa34590b4717a936f23cb95ba36a, State: Initialized, Role: FOLLOWER
I20260924 03:26:24.292385 27969 consensus_queue.cc:260] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1afa34590b4717a936f23cb95ba36a" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 43375 } } peers { permanent_uuid: "6d0793a729ba4ce799e741f50c6d532d" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45013 } } peers { permanent_uuid: "4e6fcee514da46cc9c23ce84f3fa4925" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39609 } }
I20260924 03:26:24.292799 27967 raft_consensus.cc:348] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 4e6fcee514da46cc9c23ce84f3fa4925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1afa34590b4717a936f23cb95ba36a" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 43375 } } peers { permanent_uuid: "6d0793a729ba4ce799e741f50c6d532d" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45013 } } peers { permanent_uuid: "4e6fcee514da46cc9c23ce84f3fa4925" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39609 } }
I20260924 03:26:24.293349 27967 raft_consensus.cc:374] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 4e6fcee514da46cc9c23ce84f3fa4925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:24.293649 27967 raft_consensus.cc:729] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 4e6fcee514da46cc9c23ce84f3fa4925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6fcee514da46cc9c23ce84f3fa4925, State: Initialized, Role: FOLLOWER
I20260924 03:26:24.293529 27968 raft_consensus.cc:348] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 6d0793a729ba4ce799e741f50c6d532d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1afa34590b4717a936f23cb95ba36a" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 43375 } } peers { permanent_uuid: "6d0793a729ba4ce799e741f50c6d532d" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45013 } } peers { permanent_uuid: "4e6fcee514da46cc9c23ce84f3fa4925" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39609 } }
I20260924 03:26:24.293999 27968 raft_consensus.cc:374] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 6d0793a729ba4ce799e741f50c6d532d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:24.294139 27968 raft_consensus.cc:729] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 6d0793a729ba4ce799e741f50c6d532d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d0793a729ba4ce799e741f50c6d532d, State: Initialized, Role: FOLLOWER
I20260924 03:26:24.294601 27967 consensus_queue.cc:260] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 4e6fcee514da46cc9c23ce84f3fa4925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1afa34590b4717a936f23cb95ba36a" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 43375 } } peers { permanent_uuid: "6d0793a729ba4ce799e741f50c6d532d" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45013 } } peers { permanent_uuid: "4e6fcee514da46cc9c23ce84f3fa4925" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39609 } }
I20260924 03:26:24.294739 27968 consensus_queue.cc:260] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 6d0793a729ba4ce799e741f50c6d532d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1afa34590b4717a936f23cb95ba36a" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 43375 } } peers { permanent_uuid: "6d0793a729ba4ce799e741f50c6d532d" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45013 } } peers { permanent_uuid: "4e6fcee514da46cc9c23ce84f3fa4925" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39609 } }
I20260924 03:26:24.313450 27969 ts_tablet_manager.cc:1428] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a: Time spent starting tablet: real 0.032s user 0.025s sys 0.004s
I20260924 03:26:24.313490 27814 heartbeater.cc:503] Master 127.10.12.190:33141 was elected leader, sending a full tablet report...
I20260924 03:26:24.313740 27967 ts_tablet_manager.cc:1428] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 4e6fcee514da46cc9c23ce84f3fa4925: Time spent starting tablet: real 0.032s user 0.009s sys 0.023s
I20260924 03:26:24.314028 27968 ts_tablet_manager.cc:1428] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 6d0793a729ba4ce799e741f50c6d532d: Time spent starting tablet: real 0.032s user 0.016s sys 0.016s
I20260924 03:26:24.314296 27948 heartbeater.cc:503] Master 127.10.12.190:33141 was elected leader, sending a full tablet report...
W20260924 03:26:24.349135 27815 tablet.cc:2369] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 6d0793a729ba4ce799e741f50c6d532d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:26:24.388290 27949 tablet.cc:2369] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 4e6fcee514da46cc9c23ce84f3fa4925: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:26:24.508771 27973 raft_consensus.cc:482] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:26:24.509203 27973 raft_consensus.cc:504] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1afa34590b4717a936f23cb95ba36a" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 43375 } } peers { permanent_uuid: "6d0793a729ba4ce799e741f50c6d532d" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45013 } } peers { permanent_uuid: "4e6fcee514da46cc9c23ce84f3fa4925" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39609 } }
I20260924 03:26:24.511654 27973 leader_election.cc:288] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d0793a729ba4ce799e741f50c6d532d (127.10.12.130:45013), 4e6fcee514da46cc9c23ce84f3fa4925 (127.10.12.131:39609)
I20260924 03:26:24.522109 27965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:24.512264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:51313 (local address 127.10.12.130:45013)
0924 03:26:24.512506 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:24.512513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:24.513411 (+ 898us) server_negotiation.cc:407] Connection header received
0924 03:26:24.513716 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:24.513722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:24.513809 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:24.513903 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:24.514973 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.516218 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.519407 (+ 3189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.520835 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.520992 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:24.521081 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:24.521696 (+ 615us) server_negotiation.cc:299] Negotiation successful
0924 03:26:24.521853 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 03:26:24.522152 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:24.512357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:35449 (local address 127.10.12.131:39609)
0924 03:26:24.512564 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:24.512571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:24.513343 (+ 772us) server_negotiation.cc:407] Connection header received
0924 03:26:24.513657 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:24.513664 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:24.513762 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:24.513897 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:24.514973 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.516218 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.519406 (+ 3188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.520768 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.520983 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:24.521089 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:24.521633 (+ 544us) server_negotiation.cc:299] Negotiation successful
0924 03:26:24.521799 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 03:26:24.522338 27977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:24.512072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:45013 (local address 127.10.12.129:51313)
0924 03:26:24.513160 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:24.513188 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:24.513407 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:24.513981 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:24.514056 (+ 75us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:24.514107 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:24.514812 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:24.514822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.516366 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.516372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:24.519180 (+ 2808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:24.519201 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.521139 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.521152 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:24.521435 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:24.521470 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 03:26:24.521669 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 03:26:24.521797 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"mutex_wait_us":193,"spinlock_wait_cycles":8064,"thread_start_us":674,"threads_started":1}
I20260924 03:26:24.522707 27976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:24.512189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:39609 (local address 127.10.12.129:35449)
0924 03:26:24.513197 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:24.513211 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:24.513456 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:24.513981 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:24.513987 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:24.514077 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:24.514710 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:24.514737 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.516367 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.516372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:24.519180 (+ 2808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:24.519201 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.521161 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.521167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:24.521418 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:24.521470 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 03:26:24.521584 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 03:26:24.521764 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"mutex_wait_us":285,"spinlock_wait_cycles":57088,"thread_start_us":566,"threads_started":1}
I20260924 03:26:24.523707 27769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42b1dee0f9bf4b94bafd172a4437a2c5" candidate_uuid: "9f1afa34590b4717a936f23cb95ba36a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d0793a729ba4ce799e741f50c6d532d" is_pre_election: true
I20260924 03:26:24.523747 27903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42b1dee0f9bf4b94bafd172a4437a2c5" candidate_uuid: "9f1afa34590b4717a936f23cb95ba36a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e6fcee514da46cc9c23ce84f3fa4925" is_pre_election: true
I20260924 03:26:24.524384 27903 raft_consensus.cc:2454] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 4e6fcee514da46cc9c23ce84f3fa4925 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f1afa34590b4717a936f23cb95ba36a in term 0.
I20260924 03:26:24.524384 27769 raft_consensus.cc:2454] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 6d0793a729ba4ce799e741f50c6d532d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f1afa34590b4717a936f23cb95ba36a in term 0.
I20260924 03:26:24.525173 27568 leader_election.cc:302] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e6fcee514da46cc9c23ce84f3fa4925, 9f1afa34590b4717a936f23cb95ba36a; no voters:
I20260924 03:26:24.525630 27973 raft_consensus.cc:2781] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:26:24.525806 27973 raft_consensus.cc:482] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:26:24.525926 27973 raft_consensus.cc:3037] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:24.528601 27973 raft_consensus.cc:504] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1afa34590b4717a936f23cb95ba36a" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 43375 } } peers { permanent_uuid: "6d0793a729ba4ce799e741f50c6d532d" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45013 } } peers { permanent_uuid: "4e6fcee514da46cc9c23ce84f3fa4925" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39609 } }
I20260924 03:26:24.529348 27973 leader_election.cc:288] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a [CANDIDATE]: Term 1 election: Requested vote from peers 6d0793a729ba4ce799e741f50c6d532d (127.10.12.130:45013), 4e6fcee514da46cc9c23ce84f3fa4925 (127.10.12.131:39609)
I20260924 03:26:24.530124 27769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42b1dee0f9bf4b94bafd172a4437a2c5" candidate_uuid: "9f1afa34590b4717a936f23cb95ba36a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d0793a729ba4ce799e741f50c6d532d"
I20260924 03:26:24.530138 27903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42b1dee0f9bf4b94bafd172a4437a2c5" candidate_uuid: "9f1afa34590b4717a936f23cb95ba36a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e6fcee514da46cc9c23ce84f3fa4925"
I20260924 03:26:24.530385 27769 raft_consensus.cc:3037] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 6d0793a729ba4ce799e741f50c6d532d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:24.530432 27903 raft_consensus.cc:3037] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 4e6fcee514da46cc9c23ce84f3fa4925 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:24.533349 27769 raft_consensus.cc:2454] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 6d0793a729ba4ce799e741f50c6d532d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1afa34590b4717a936f23cb95ba36a in term 1.
I20260924 03:26:24.533367 27903 raft_consensus.cc:2454] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 4e6fcee514da46cc9c23ce84f3fa4925 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1afa34590b4717a936f23cb95ba36a in term 1.
I20260924 03:26:24.533969 27568 leader_election.cc:302] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e6fcee514da46cc9c23ce84f3fa4925, 9f1afa34590b4717a936f23cb95ba36a; no voters:
I20260924 03:26:24.534339 27973 raft_consensus.cc:2781] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:26:24.534934 27973 raft_consensus.cc:686] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a [term 1 LEADER]: Becoming Leader. State: Replica: 9f1afa34590b4717a936f23cb95ba36a, State: Running, Role: LEADER
I20260924 03:26:24.535871 27973 consensus_queue.cc:237] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1afa34590b4717a936f23cb95ba36a" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 43375 } } peers { permanent_uuid: "6d0793a729ba4ce799e741f50c6d532d" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45013 } } peers { permanent_uuid: "4e6fcee514da46cc9c23ce84f3fa4925" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39609 } }
I20260924 03:26:24.541733 27490 catalog_manager.cc:5495] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f1afa34590b4717a936f23cb95ba36a (127.10.12.129). New cstate: current_term: 1 leader_uuid: "9f1afa34590b4717a936f23cb95ba36a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1afa34590b4717a936f23cb95ba36a" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 43375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d0793a729ba4ce799e741f50c6d532d" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e6fcee514da46cc9c23ce84f3fa4925" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39609 } health_report { overall_health: UNKNOWN } } }
W20260924 03:26:24.555853 27681 tablet.cc:2369] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:26:24.627878 27963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:24.621163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43796 (local address 127.10.12.129:43375)
0924 03:26:24.621456 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:24.621461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:24.621481 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:26:24.621608 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:24.621613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:24.621723 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:24.621884 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:26:24.622375 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.623311 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.625220 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.625727 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.625845 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:24.626538 (+ 693us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:26:24.627277 (+ 739us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:26:24.627408 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:24.627517 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 03:26:24.627613 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260924 03:26:24.639883 27903 raft_consensus.cc:1261] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 4e6fcee514da46cc9c23ce84f3fa4925 [term 1 FOLLOWER]: Refusing update from remote peer 9f1afa34590b4717a936f23cb95ba36a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:26:24.639883 27769 raft_consensus.cc:1261] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 6d0793a729ba4ce799e741f50c6d532d [term 1 FOLLOWER]: Refusing update from remote peer 9f1afa34590b4717a936f23cb95ba36a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:26:24.641191 27978 consensus_queue.cc:1035] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6fcee514da46cc9c23ce84f3fa4925" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:26:24.641677 27973 consensus_queue.cc:1035] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d0793a729ba4ce799e741f50c6d532d" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:26:24.656729 27983 mvcc.cc:205] Tried to move back new op lower bound from 7332742695469350912 to 7332742695064330240. Current Snapshot: MvccSnapshot[applied={T|T < 7332742695469350912}]
I20260924 03:26:24.674293 27490 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:26:24.674635 27490 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:26:24.679394 27490 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e308e5ef9e144d5ba0d88c7d170333d: Sending DeleteTablet for 3 replicas of tablet 42b1dee0f9bf4b94bafd172a4437a2c5
I20260924 03:26:24.680991 27749 tablet_service.cc:1514] Processing DeleteTablet for tablet 42b1dee0f9bf4b94bafd172a4437a2c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:26:24 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:59332
I20260924 03:26:24.680979 27615 tablet_service.cc:1514] Processing DeleteTablet for tablet 42b1dee0f9bf4b94bafd172a4437a2c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:26:24 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:43782
I20260924 03:26:24.681711 27883 tablet_service.cc:1514] Processing DeleteTablet for tablet 42b1dee0f9bf4b94bafd172a4437a2c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:26:24 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:60982
I20260924 03:26:24.683719 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 27552
I20260924 03:26:24.693462 27992 tablet_replica.cc:330] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a: stopping tablet replica
I20260924 03:26:24.694329 27993 tablet_replica.cc:330] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 4e6fcee514da46cc9c23ce84f3fa4925: stopping tablet replica
I20260924 03:26:24.694959 27993 raft_consensus.cc:2229] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 4e6fcee514da46cc9c23ce84f3fa4925 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:26:24.694914 27992 raft_consensus.cc:2229] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:26:24.695782 27993 raft_consensus.cc:2258] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 4e6fcee514da46cc9c23ce84f3fa4925 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:26:24.695820 27992 raft_consensus.cc:2258] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:26:24.699275 27993 ts_tablet_manager.cc:1905] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 4e6fcee514da46cc9c23ce84f3fa4925: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:26:24.700057 27991 tablet_replica.cc:330] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 6d0793a729ba4ce799e741f50c6d532d: stopping tablet replica
I20260924 03:26:24.700647 27991 raft_consensus.cc:2229] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 6d0793a729ba4ce799e741f50c6d532d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:26:24.701331 27991 raft_consensus.cc:2258] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 6d0793a729ba4ce799e741f50c6d532d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:26:24.701921 27992 ts_tablet_manager.cc:1905] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:26:24.704026 27991 ts_tablet_manager.cc:1905] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 6d0793a729ba4ce799e741f50c6d532d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:26:24.708833 27993 ts_tablet_manager.cc:1918] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 4e6fcee514da46cc9c23ce84f3fa4925: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:26:24.709233 27993 log.cc:1196] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 4e6fcee514da46cc9c23ce84f3fa4925: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/42b1dee0f9bf4b94bafd172a4437a2c5
I20260924 03:26:24.709944 27993 ts_tablet_manager.cc:1939] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 4e6fcee514da46cc9c23ce84f3fa4925: Deleting consensus metadata
Sep 24 03:26:24 dist-test-slave-00qs krb5kdc[27446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.129@KRBTEST.COM
I20260924 03:26:24.712296 27991 ts_tablet_manager.cc:1918] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 6d0793a729ba4ce799e741f50c6d532d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:26:24.712558 27991 log.cc:1196] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 6d0793a729ba4ce799e741f50c6d532d: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/42b1dee0f9bf4b94bafd172a4437a2c5
I20260924 03:26:24.713222 27991 ts_tablet_manager.cc:1939] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 6d0793a729ba4ce799e741f50c6d532d: Deleting consensus metadata
I20260924 03:26:24.713559 27475 catalog_manager.cc:4841] TS 4e6fcee514da46cc9c23ce84f3fa4925 (127.10.12.131:39609): tablet 42b1dee0f9bf4b94bafd172a4437a2c5 (table test-table [id=3c16b840075a4812aa77d56c094dbdbf]) successfully deleted
I20260924 03:26:24.715758 27478 catalog_manager.cc:4841] TS 6d0793a729ba4ce799e741f50c6d532d (127.10.12.130:45013): tablet 42b1dee0f9bf4b94bafd172a4437a2c5 (table test-table [id=3c16b840075a4812aa77d56c094dbdbf]) successfully deleted
I20260924 03:26:24.717322 27992 ts_tablet_manager.cc:1918] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:26:24.717555 27992 log.cc:1196] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790220067268388-10290-0/minicluster-data/ts-0/wal/wals/42b1dee0f9bf4b94bafd172a4437a2c5
I20260924 03:26:24.718292 27992 ts_tablet_manager.cc:1939] T 42b1dee0f9bf4b94bafd172a4437a2c5 P 9f1afa34590b4717a936f23cb95ba36a: Deleting consensus metadata
I20260924 03:26:24.720978 27476 catalog_manager.cc:4841] TS 9f1afa34590b4717a936f23cb95ba36a (127.10.12.129:43375): tablet 42b1dee0f9bf4b94bafd172a4437a2c5 (table test-table [id=3c16b840075a4812aa77d56c094dbdbf]) successfully deleted
I20260924 03:26:24.723472 27963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:24.688207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43798 (local address 127.10.12.129:43375)
0924 03:26:24.688461 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:24.688469 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:24.691929 (+ 3460us) server_negotiation.cc:407] Connection header received
0924 03:26:24.692333 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:24.692341 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:24.692468 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:24.692604 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:24.695321 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.696917 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.702222 (+ 5305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.703059 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.703210 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:24.715794 (+ 12584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:24.715817 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:24.715841 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:24.715911 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:24.719931 (+ 4020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:24.720924 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:24.720936 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:24.720940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:24.721166 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:24.721757 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:24.721761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:24.721764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:24.722213 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:24.722501 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:24.722904 (+ 403us) server_negotiation.cc:299] Negotiation successful
0924 03:26:24.723132 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260924 03:26:24.724624 27675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:24.839430 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 27552
I20260924 03:26:24.858857 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 27686
Sep 24 03:26:24 dist-test-slave-00qs krb5kdc[27446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.130@KRBTEST.COM
I20260924 03:26:24.878832 27965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:24.860363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59342 (local address 127.10.12.130:45013)
0924 03:26:24.860597 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:24.860602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:24.860631 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:26:24.860706 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:24.860712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:24.860819 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:24.860932 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:24.862495 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.863340 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.865237 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:24.865744 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:24.865945 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:24.871294 (+ 5349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:24.871323 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:24.871345 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:24.871424 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:24.874899 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:24.876144 (+ 1245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:24.876155 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:24.876158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:24.876357 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:24.876868 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:24.876873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:24.876876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:24.877407 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:24.877714 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:24.878118 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 03:26:24.878366 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260924 03:26:24.879741 27809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:24.979821 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 27686
I20260924 03:26:24.996692 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 27820
Sep 24 03:26:25 dist-test-slave-00qs krb5kdc[27446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.131@KRBTEST.COM
I20260924 03:26:25.017935 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:24.997718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60986 (local address 127.10.12.131:39609)
0924 03:26:24.997927 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:24.997933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:24.997953 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:26:24.998172 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:24.998178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:24.998263 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:24.998364 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:25.000018 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:25.000888 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:25.002599 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:25.003102 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:25.003236 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:25.009123 (+ 5887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:25.009146 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:25.009167 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:25.009234 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:25.013167 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:25.014457 (+ 1290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:25.014470 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:25.014473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:25.014809 (+ 336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:25.015399 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:25.015404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:25.015407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:25.016030 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:25.016401 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:25.017163 (+ 762us) server_negotiation.cc:299] Negotiation successful
0924 03:26:25.017453 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 03:26:25.018924 27943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:25.121456 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 27820
I20260924 03:26:25.137923 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 27459
I20260924 03:26:25.139928 27521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:25.269348 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 27459
2026-09-24T03:26:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4742 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T03:26:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:26:25Z Disabled control of system clock
I20260924 03:26:25.358075 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:34759
--webserver_interface=127.10.12.190
--webserver_port=0
--builtin_ntp_servers=127.10.12.148:46561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:34759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:26:25.710093 28002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:25.710436 28002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:25.710549 28002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:25.718398 28002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:26:25.718528 28002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:25.718568 28002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:25.718648 28002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:26:25.718706 28002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:26:25.728585 28002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:46561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:34759
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:34759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:25.730067 28002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:25.731899 28002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:25.744248 28008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:25.744318 28007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:25.744304 28010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:25.744964 28002 server_base.cc:1034] running on GCE node
I20260924 03:26:25.745806 28002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:25.747289 28002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:25.748522 28002 hybrid_clock.cc:648] HybridClock initialized: now 1790220385748475 us; error 38 us; skew 500 ppm
I20260924 03:26:25.748904 28002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:26:25.758288 28002 webserver.cc:460] Webserver started at http://127.10.12.190:33627/ using document root <none> and password file <none>
I20260924 03:26:25.759112 28002 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:25.759220 28002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:25.759546 28002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:25.762696 28002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "b14fdde53574424893dc719f31b8cff0"
format_stamp: "Formatted at 2026-09-24 03:26:25 on dist-test-slave-00qs"
server_key: "55c023499c3bc839a1cdbe9a9ee45aa4"
server_key_iv: "098794ea968a0a9c784ec734c01ff38e"
server_key_version: "encryptionkey@0"
I20260924 03:26:25.763567 28002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "b14fdde53574424893dc719f31b8cff0"
format_stamp: "Formatted at 2026-09-24 03:26:25 on dist-test-slave-00qs"
server_key: "55c023499c3bc839a1cdbe9a9ee45aa4"
server_key_iv: "098794ea968a0a9c784ec734c01ff38e"
server_key_version: "encryptionkey@0"
I20260924 03:26:25.769160 28002 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 03:26:25.773192 28016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:25.774622 28002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260924 03:26:25.774803 28002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "b14fdde53574424893dc719f31b8cff0"
format_stamp: "Formatted at 2026-09-24 03:26:25 on dist-test-slave-00qs"
server_key: "55c023499c3bc839a1cdbe9a9ee45aa4"
server_key_iv: "098794ea968a0a9c784ec734c01ff38e"
server_key_version: "encryptionkey@0"
I20260924 03:26:25.775024 28002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:25.813591 28002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:25.814531 28002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:25.814832 28002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:25.833760 28067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:34759 every 8 connection(s)
I20260924 03:26:25.833752 28002 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:34759
I20260924 03:26:25.835263 28002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:26:25.838395 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 28002
I20260924 03:26:25.838579 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:26:25.839154 10290 external_mini_cluster.cc:1428] Setting key 7fea0963b611e2138be794b0b4ce708e
I20260924 03:26:25.839500 28068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:25.852898 28068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b14fdde53574424893dc719f31b8cff0: Bootstrap starting.
I20260924 03:26:25.853791 28071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:25.841252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42958 (local address 127.10.12.190:34759)
0924 03:26:25.842270 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:25.842281 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:25.842327 (+ 46us) server_negotiation.cc:407] Connection header received
0924 03:26:25.844528 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:25.844554 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:25.845069 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:25.845504 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:25.846120 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:25.847604 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:25.849120 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:25.849830 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:25.850038 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:25.851230 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:25.851238 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:25.851254 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:25.851815 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:25.851923 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:25.851946 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:25.852098 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:25.852320 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 03:26:25.852737 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":359,"threads_started":1}
I20260924 03:26:25.856320 28068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b14fdde53574424893dc719f31b8cff0: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:25.857415 28068 log.cc:824] T 00000000000000000000000000000000 P b14fdde53574424893dc719f31b8cff0: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:25.860452 28068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b14fdde53574424893dc719f31b8cff0: No bootstrap required, opened a new log
I20260924 03:26:25.868811 28068 raft_consensus.cc:348] T 00000000000000000000000000000000 P b14fdde53574424893dc719f31b8cff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b14fdde53574424893dc719f31b8cff0" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 34759 } }
I20260924 03:26:25.869249 28068 raft_consensus.cc:374] T 00000000000000000000000000000000 P b14fdde53574424893dc719f31b8cff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:25.869354 28068 raft_consensus.cc:729] T 00000000000000000000000000000000 P b14fdde53574424893dc719f31b8cff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b14fdde53574424893dc719f31b8cff0, State: Initialized, Role: FOLLOWER
I20260924 03:26:25.869993 28068 consensus_queue.cc:260] T 00000000000000000000000000000000 P b14fdde53574424893dc719f31b8cff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b14fdde53574424893dc719f31b8cff0" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 34759 } }
I20260924 03:26:25.870262 28068 raft_consensus.cc:388] T 00000000000000000000000000000000 P b14fdde53574424893dc719f31b8cff0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:26:25.870478 28068 raft_consensus.cc:482] T 00000000000000000000000000000000 P b14fdde53574424893dc719f31b8cff0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:26:25.870649 28068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b14fdde53574424893dc719f31b8cff0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:25.873286 28068 raft_consensus.cc:504] T 00000000000000000000000000000000 P b14fdde53574424893dc719f31b8cff0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b14fdde53574424893dc719f31b8cff0" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 34759 } }
I20260924 03:26:25.873999 28068 leader_election.cc:302] T 00000000000000000000000000000000 P b14fdde53574424893dc719f31b8cff0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b14fdde53574424893dc719f31b8cff0; no voters:
I20260924 03:26:25.874653 28068 leader_election.cc:288] T 00000000000000000000000000000000 P b14fdde53574424893dc719f31b8cff0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:26:25.874812 28073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b14fdde53574424893dc719f31b8cff0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:26:25.875881 28073 raft_consensus.cc:686] T 00000000000000000000000000000000 P b14fdde53574424893dc719f31b8cff0 [term 1 LEADER]: Becoming Leader. State: Replica: b14fdde53574424893dc719f31b8cff0, State: Running, Role: LEADER
I20260924 03:26:25.876706 28073 consensus_queue.cc:237] T 00000000000000000000000000000000 P b14fdde53574424893dc719f31b8cff0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b14fdde53574424893dc719f31b8cff0" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 34759 } }
I20260924 03:26:25.876945 28068 sys_catalog.cc:564] T 00000000000000000000000000000000 P b14fdde53574424893dc719f31b8cff0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:26:25.879755 28075 sys_catalog.cc:455] T 00000000000000000000000000000000 P b14fdde53574424893dc719f31b8cff0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b14fdde53574424893dc719f31b8cff0. Latest consensus state: current_term: 1 leader_uuid: "b14fdde53574424893dc719f31b8cff0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b14fdde53574424893dc719f31b8cff0" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 34759 } } }
I20260924 03:26:25.880256 28075 sys_catalog.cc:458] T 00000000000000000000000000000000 P b14fdde53574424893dc719f31b8cff0 [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:25.880100 28074 sys_catalog.cc:455] T 00000000000000000000000000000000 P b14fdde53574424893dc719f31b8cff0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b14fdde53574424893dc719f31b8cff0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b14fdde53574424893dc719f31b8cff0" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 34759 } } }
I20260924 03:26:25.880411 28074 sys_catalog.cc:458] T 00000000000000000000000000000000 P b14fdde53574424893dc719f31b8cff0 [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:25.882526 28081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:26:25.888360 28081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:26:25.898702 28081 catalog_manager.cc:1347] Generated new cluster ID: f5822eb6848b4a7da32c4e2f642f46d6
I20260924 03:26:25.898836 28081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:26:25.909356 28081 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:26:25.910746 28081 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:26:25.922119 28081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b14fdde53574424893dc719f31b8cff0: Generated new TSK 0
I20260924 03:26:25.923072 28081 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:26:25.939465 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:34759
--builtin_ntp_servers=127.10.12.148:46561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:26:26.297858 28092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:26.298162 28092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:26.298281 28092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:26.305899 28092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:26.306022 28092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:26.306169 28092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:26:26.316313 28092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:46561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:34759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:26.317589 28092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:26.320463 28092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:26.333956 28097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:26.334046 28098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:26.333997 28100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:26.334950 28092 server_base.cc:1034] running on GCE node
I20260924 03:26:26.335533 28092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:26.336585 28092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:26.337812 28092 hybrid_clock.cc:648] HybridClock initialized: now 1790220386337734 us; error 60 us; skew 500 ppm
I20260924 03:26:26.338164 28092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:26:26.340975 28092 webserver.cc:460] Webserver started at http://127.10.12.129:38925/ using document root <none> and password file <none>
I20260924 03:26:26.341723 28092 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:26.341828 28092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:26.342152 28092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:26.345155 28092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "993551e384ea4d85832e5a3affa6fec9"
format_stamp: "Formatted at 2026-09-24 03:26:26 on dist-test-slave-00qs"
server_key: "c032b11fb80ffc1e10b2114abbc9b5de"
server_key_iv: "fca7082b509c199eeb415c878049fb75"
server_key_version: "encryptionkey@0"
I20260924 03:26:26.345954 28092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "993551e384ea4d85832e5a3affa6fec9"
format_stamp: "Formatted at 2026-09-24 03:26:26 on dist-test-slave-00qs"
server_key: "c032b11fb80ffc1e10b2114abbc9b5de"
server_key_iv: "fca7082b509c199eeb415c878049fb75"
server_key_version: "encryptionkey@0"
I20260924 03:26:26.351150 28092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 03:26:26.354838 28106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:26.356438 28092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:26:26.356869 28092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "993551e384ea4d85832e5a3affa6fec9"
format_stamp: "Formatted at 2026-09-24 03:26:26 on dist-test-slave-00qs"
server_key: "c032b11fb80ffc1e10b2114abbc9b5de"
server_key_iv: "fca7082b509c199eeb415c878049fb75"
server_key_version: "encryptionkey@0"
I20260924 03:26:26.357251 28092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:26.377897 28092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:26.378845 28092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:26.379148 28092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:26.380254 28092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:26:26.381915 28092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:26:26.382017 28092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:26.382130 28092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:26:26.382200 28092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:26.415741 28092 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:37377
I20260924 03:26:26.415789 28218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:37377 every 8 connection(s)
I20260924 03:26:26.417084 28092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:26:26.417941 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 28092
I20260924 03:26:26.418133 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:26:26.418751 10290 external_mini_cluster.cc:1428] Setting key ea189b359225d6343a983b6091e39ff4
I20260924 03:26:26.422091 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:34759
--builtin_ntp_servers=127.10.12.148:46561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:26:26.432935 28221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:26.420062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:40931 (local address 127.10.12.190:34759)
0924 03:26:26.420588 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:26.420595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:26.422073 (+ 1478us) server_negotiation.cc:407] Connection header received
0924 03:26:26.424570 (+ 2497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:26.424574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:26.424649 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:26.424747 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:26.426351 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:26.427291 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:26.429734 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:26.430225 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:26.430391 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:26.432048 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:26.432054 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:26.432057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:26.432080 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:26.432119 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:26.432132 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:26.432212 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:26.432442 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 03:26:26.432552 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":295,"threads_started":1}
I20260924 03:26:26.434051 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:26.420775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:34759 (local address 127.10.12.129:40931)
0924 03:26:26.421894 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:26.421953 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:26.424248 (+ 2295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:26.425040 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:26.425052 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:26.425177 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:26:26.425996 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:26.426021 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:26.427500 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:26.427506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:26.429498 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:26.429518 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:26.430508 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:26.430516 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:26.430766 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:26.431645 (+ 879us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:26:26.431648 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:26:26.431695 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:26:26.431789 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:26:26.432228 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:26:26.432240 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:26:26.432249 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 03:26:26.432417 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 03:26:26.432896 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":304,"threads_started":1}
I20260924 03:26:26.435582 28219 heartbeater.cc:347] Connected to a master server at 127.10.12.190:34759
I20260924 03:26:26.435933 28219 heartbeater.cc:464] Registering TS with master...
I20260924 03:26:26.436921 28219 heartbeater.cc:511] Master 127.10.12.190:34759 requested a full tablet report, sending...
I20260924 03:26:26.439337 28033 ts_manager.cc:194] Registered new tserver with Master: 993551e384ea4d85832e5a3affa6fec9 (127.10.12.129:37377)
I20260924 03:26:26.440892 28033 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.12.129:40931
W20260924 03:26:26.764071 28223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:26.764365 28223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:26.764479 28223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:26.772238 28223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:26.772360 28223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:26.772508 28223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:26:26.782220 28223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:46561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:34759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:26.783607 28223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:26.785439 28223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:26.799225 28228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:26.799199 28229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:26.799211 28231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:26.799815 28223 server_base.cc:1034] running on GCE node
I20260924 03:26:26.800514 28223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:26.801509 28223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:26.802734 28223 hybrid_clock.cc:648] HybridClock initialized: now 1790220386802672 us; error 55 us; skew 500 ppm
I20260924 03:26:26.803086 28223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:26:26.805466 28223 webserver.cc:460] Webserver started at http://127.10.12.130:33757/ using document root <none> and password file <none>
I20260924 03:26:26.806223 28223 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:26.806319 28223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:26.806629 28223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:26.809670 28223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "0bf3c1db9f574312be26a2ee04550de6"
format_stamp: "Formatted at 2026-09-24 03:26:26 on dist-test-slave-00qs"
server_key: "330b1e24e945d9cdb3203578e95e622f"
server_key_iv: "542a3741963371a0c801e8600bdd33f6"
server_key_version: "encryptionkey@0"
I20260924 03:26:26.810480 28223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "0bf3c1db9f574312be26a2ee04550de6"
format_stamp: "Formatted at 2026-09-24 03:26:26 on dist-test-slave-00qs"
server_key: "330b1e24e945d9cdb3203578e95e622f"
server_key_iv: "542a3741963371a0c801e8600bdd33f6"
server_key_version: "encryptionkey@0"
I20260924 03:26:26.815824 28223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 03:26:26.819710 28237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:26.821199 28223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:26:26.821381 28223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "0bf3c1db9f574312be26a2ee04550de6"
format_stamp: "Formatted at 2026-09-24 03:26:26 on dist-test-slave-00qs"
server_key: "330b1e24e945d9cdb3203578e95e622f"
server_key_iv: "542a3741963371a0c801e8600bdd33f6"
server_key_version: "encryptionkey@0"
I20260924 03:26:26.821602 28223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:26.847652 28223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:26.848564 28223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:26.848873 28223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:26.849918 28223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:26:26.851445 28223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:26:26.851547 28223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:26.851652 28223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:26:26.851724 28223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:26.886935 28223 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:44577
I20260924 03:26:26.887009 28349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:44577 every 8 connection(s)
I20260924 03:26:26.888260 28223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:26:26.891010 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 28223
I20260924 03:26:26.891201 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:26:26.891734 10290 external_mini_cluster.cc:1428] Setting key 1921340ec36ff3e7990a1f52c3744805
I20260924 03:26:26.895167 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:34759
--builtin_ntp_servers=127.10.12.148:46561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:26:26.902945 28221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:26.890953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:41681 (local address 127.10.12.190:34759)
0924 03:26:26.891170 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:26.891177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:26.892879 (+ 1702us) server_negotiation.cc:407] Connection header received
0924 03:26:26.895380 (+ 2501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:26.895387 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:26.895484 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:26.895588 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:26.897149 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:26.897876 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:26.899944 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:26.900450 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:26.900552 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:26.902252 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:26.902257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:26.902260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:26.902286 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:26.902320 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:26.902332 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:26.902404 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:26.902620 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 03:26:26.902709 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 03:26:26.904327 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:26.891602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:34759 (local address 127.10.12.130:41681)
0924 03:26:26.892691 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:26.892746 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:26.895022 (+ 2276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:26.895914 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:26.895927 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:26.896096 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:26:26.896926 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:26.896949 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:26.898034 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:26.898040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:26.899769 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:26.899785 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:26.900726 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:26.900734 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:26.901055 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:26.901851 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:26:26.901854 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:26:26.901915 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:26:26.902024 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:26:26.902419 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:26:26.902427 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:26:26.902435 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 03:26:26.902611 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 03:26:26.903046 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":352,"threads_started":1}
I20260924 03:26:26.905947 28350 heartbeater.cc:347] Connected to a master server at 127.10.12.190:34759
I20260924 03:26:26.906250 28350 heartbeater.cc:464] Registering TS with master...
I20260924 03:26:26.907209 28350 heartbeater.cc:511] Master 127.10.12.190:34759 requested a full tablet report, sending...
I20260924 03:26:26.908828 28033 ts_manager.cc:194] Registered new tserver with Master: 0bf3c1db9f574312be26a2ee04550de6 (127.10.12.130:44577)
I20260924 03:26:26.909873 28033 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.12.130:41681
W20260924 03:26:27.236534 28353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:27.236829 28353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:27.236917 28353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:27.244520 28353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:27.244643 28353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:27.244755 28353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:26:27.254144 28353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:46561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:34759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:27.255381 28353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:27.257153 28353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:27.271026 28361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:27.271139 28358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:27.271096 28359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:27.271804 28353 server_base.cc:1034] running on GCE node
I20260924 03:26:27.272589 28353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:27.273887 28353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:27.275148 28353 hybrid_clock.cc:648] HybridClock initialized: now 1790220387275074 us; error 65 us; skew 500 ppm
I20260924 03:26:27.275514 28353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:26:27.278122 28353 webserver.cc:460] Webserver started at http://127.10.12.131:38585/ using document root <none> and password file <none>
I20260924 03:26:27.278952 28353 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:27.279062 28353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:27.279371 28353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:27.282567 28353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "15348a4c611047f39df72f27d07d37d3"
format_stamp: "Formatted at 2026-09-24 03:26:27 on dist-test-slave-00qs"
server_key: "720243d337889be56f42c3d8d4d40ad3"
server_key_iv: "bf5549e7f200581a333711f144e9d236"
server_key_version: "encryptionkey@0"
I20260924 03:26:27.283401 28353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "15348a4c611047f39df72f27d07d37d3"
format_stamp: "Formatted at 2026-09-24 03:26:27 on dist-test-slave-00qs"
server_key: "720243d337889be56f42c3d8d4d40ad3"
server_key_iv: "bf5549e7f200581a333711f144e9d236"
server_key_version: "encryptionkey@0"
I20260924 03:26:27.289110 28353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 03:26:27.293263 28367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:27.294831 28353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:26:27.295019 28353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "15348a4c611047f39df72f27d07d37d3"
format_stamp: "Formatted at 2026-09-24 03:26:27 on dist-test-slave-00qs"
server_key: "720243d337889be56f42c3d8d4d40ad3"
server_key_iv: "bf5549e7f200581a333711f144e9d236"
server_key_version: "encryptionkey@0"
I20260924 03:26:27.295240 28353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:27.319411 28353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:27.320400 28353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:27.320704 28353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:27.321720 28353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:26:27.323251 28353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:26:27.323350 28353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:27.323464 28353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:26:27.323535 28353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:27.355792 28353 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:39927
I20260924 03:26:27.355862 28479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:39927 every 8 connection(s)
I20260924 03:26:27.357093 28353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:26:27.364393 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 28353
I20260924 03:26:27.364598 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:26:27.365196 10290 external_mini_cluster.cc:1428] Setting key 582869f91da2b1cf4568e9f2fefe20f9
I20260924 03:26:27.370755 28221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:27.359754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:57831 (local address 127.10.12.190:34759)
0924 03:26:27.360005 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:27.360011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:27.361539 (+ 1528us) server_negotiation.cc:407] Connection header received
0924 03:26:27.363449 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:27.363456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:27.363560 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:27.363684 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:27.365215 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.366107 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.367984 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.368451 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.368556 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:27.369920 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:27.369926 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:27.369929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:27.369950 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:27.369990 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:27.370003 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:27.370082 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:27.370396 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 03:26:27.370490 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 03:26:27.371809 28482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:27.360333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:34759 (local address 127.10.12.131:57831)
0924 03:26:27.361327 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:27.361380 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:27.363093 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:27.363944 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:27.363987 (+ 43us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:27.364123 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:26:27.365010 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:27.365042 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.366274 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.366279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:27.367815 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:27.367834 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.368671 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.368676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:27.368892 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:27.369576 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:26:27.369578 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:26:27.369621 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:26:27.369716 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:26:27.370126 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:26:27.370136 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:26:27.370143 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 03:26:27.370377 (+ 234us) client_negotiation.cc:240] Negotiation successful
0924 03:26:27.370707 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":315,"threads_started":1}
I20260924 03:26:27.373499 28480 heartbeater.cc:347] Connected to a master server at 127.10.12.190:34759
I20260924 03:26:27.374058 28480 heartbeater.cc:464] Registering TS with master...
I20260924 03:26:27.375100 28480 heartbeater.cc:511] Master 127.10.12.190:34759 requested a full tablet report, sending...
I20260924 03:26:27.377221 28033 ts_manager.cc:194] Registered new tserver with Master: 15348a4c611047f39df72f27d07d37d3 (127.10.12.131:39927)
I20260924 03:26:27.378216 28033 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.12.131:57831
I20260924 03:26:27.381330 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:26:27.392436 28221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:27.386412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42968 (local address 127.10.12.190:34759)
0924 03:26:27.386652 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:27.386659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:27.386933 (+ 274us) server_negotiation.cc:407] Connection header received
0924 03:26:27.387060 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:27.387065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:27.387148 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:27.387239 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:27.387683 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.388480 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.390018 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.390490 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.390604 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:27.391765 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:27.391771 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:27.391774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:27.391803 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:27.391839 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:27.391852 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:27.391924 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:27.392098 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 03:26:27.392194 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 03:26:27.400676 28033 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:26:27.407617 28033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:26:27.433697 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:27.420949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:39927 (local address 127.0.0.1:42858)
0924 03:26:27.421803 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:27.421856 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:27.422055 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:27.424099 (+ 2044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:27.424108 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:27.424144 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:27.424666 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:27.424680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.427375 (+ 2695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.427382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:27.430059 (+ 2677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:27.430085 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.432548 (+ 2463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.432558 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:27.432832 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:27.432918 (+ 86us) client_negotiation.cc:769] Sending connection context
0924 03:26:27.433055 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 03:26:27.433292 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":342,"threads_started":1}
I20260924 03:26:27.435326 28493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:27.421101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42858 (local address 127.10.12.131:39927)
0924 03:26:27.423296 (+ 2195us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:27.423316 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:27.423356 (+ 40us) server_negotiation.cc:407] Connection header received
0924 03:26:27.423504 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:27.423513 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:27.423790 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:27.424996 (+ 1206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:27.425304 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.427174 (+ 1870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.430422 (+ 3248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.432083 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.434356 (+ 2273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:27.434718 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:27.434796 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 03:26:27.434994 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2000,"thread_start_us":1874,"threads_started":1}
I20260924 03:26:27.440526 28415 tablet_service.cc:1467] Processing CreateTablet for tablet 69172f8ce3bc43cf8983def8601a8014 (DEFAULT_TABLE table=test-table [id=b28bdedcc4ed49eea087004744c242b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:26:27.441893 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:27.421608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:44577 (local address 127.0.0.1:34424)
0924 03:26:27.422627 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:27.422645 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:27.422756 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:27.434396 (+ 11640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:27.434402 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:27.434437 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:27.434806 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:27.434818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.437072 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.437078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:27.439104 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:27.439122 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.440994 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.441000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:27.441200 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:27.441236 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 03:26:27.441330 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 03:26:27.441480 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"mutex_wait_us":4,"thread_start_us":760,"threads_started":1}
I20260924 03:26:27.442082 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:27.422781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56118 (local address 127.10.12.129:37377)
0924 03:26:27.435181 (+ 12400us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:27.435189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:27.435218 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:26:27.435317 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:27.435325 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:27.435581 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:27.436224 (+ 643us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:27.436473 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.437726 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.439830 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.440796 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.441227 (+ 431us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:27.441503 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:27.441577 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 03:26:27.441742 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12213,"thread_start_us":12814,"threads_started":1}
I20260924 03:26:27.442420 28415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69172f8ce3bc43cf8983def8601a8014. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:27.442413 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:27.422291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:37377 (local address 127.0.0.1:56118)
0924 03:26:27.434037 (+ 11746us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:27.434060 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:27.434179 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:27.435821 (+ 1642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:27.435826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:27.435855 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:27.436045 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:27.436056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.437893 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.437897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:27.439644 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:27.439659 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.440983 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.440989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:27.441354 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:27.441391 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 03:26:27.441470 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 03:26:27.441616 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11621,"mutex_wait_us":144,"thread_start_us":13277,"threads_started":1}
I20260924 03:26:27.442703 28495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:27.422185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34424 (local address 127.10.12.130:44577)
0924 03:26:27.433392 (+ 11207us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:27.433403 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:27.433443 (+ 40us) server_negotiation.cc:407] Connection header received
0924 03:26:27.433568 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:27.433578 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:27.433895 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:27.435031 (+ 1136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:27.435299 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.436934 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.439356 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.440763 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.440915 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:27.441155 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:27.442189 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0924 03:26:27.442365 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10967,"thread_start_us":11874,"threads_started":1}
I20260924 03:26:27.444772 28219 heartbeater.cc:503] Master 127.10.12.190:34759 was elected leader, sending a full tablet report...
I20260924 03:26:27.446926 28154 tablet_service.cc:1467] Processing CreateTablet for tablet 69172f8ce3bc43cf8983def8601a8014 (DEFAULT_TABLE table=test-table [id=b28bdedcc4ed49eea087004744c242b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:26:27.446935 28285 tablet_service.cc:1467] Processing CreateTablet for tablet 69172f8ce3bc43cf8983def8601a8014 (DEFAULT_TABLE table=test-table [id=b28bdedcc4ed49eea087004744c242b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:26:27.448442 28285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69172f8ce3bc43cf8983def8601a8014. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:27.448505 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69172f8ce3bc43cf8983def8601a8014. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:27.457967 28498 tablet_bootstrap.cc:492] T 69172f8ce3bc43cf8983def8601a8014 P 15348a4c611047f39df72f27d07d37d3: Bootstrap starting.
I20260924 03:26:27.462824 28498 tablet_bootstrap.cc:654] T 69172f8ce3bc43cf8983def8601a8014 P 15348a4c611047f39df72f27d07d37d3: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:27.463220 28500 tablet_bootstrap.cc:492] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9: Bootstrap starting.
I20260924 03:26:27.464617 28498 log.cc:824] T 69172f8ce3bc43cf8983def8601a8014 P 15348a4c611047f39df72f27d07d37d3: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:27.464931 28499 tablet_bootstrap.cc:492] T 69172f8ce3bc43cf8983def8601a8014 P 0bf3c1db9f574312be26a2ee04550de6: Bootstrap starting.
I20260924 03:26:27.467831 28500 tablet_bootstrap.cc:654] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:27.468384 28499 tablet_bootstrap.cc:654] T 69172f8ce3bc43cf8983def8601a8014 P 0bf3c1db9f574312be26a2ee04550de6: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:27.469710 28499 log.cc:824] T 69172f8ce3bc43cf8983def8601a8014 P 0bf3c1db9f574312be26a2ee04550de6: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:27.469872 28500 log.cc:824] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:27.474516 28498 tablet_bootstrap.cc:492] T 69172f8ce3bc43cf8983def8601a8014 P 15348a4c611047f39df72f27d07d37d3: No bootstrap required, opened a new log
I20260924 03:26:27.474891 28498 ts_tablet_manager.cc:1397] T 69172f8ce3bc43cf8983def8601a8014 P 15348a4c611047f39df72f27d07d37d3: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.002s
I20260924 03:26:27.484797 28499 tablet_bootstrap.cc:492] T 69172f8ce3bc43cf8983def8601a8014 P 0bf3c1db9f574312be26a2ee04550de6: No bootstrap required, opened a new log
I20260924 03:26:27.485127 28499 ts_tablet_manager.cc:1397] T 69172f8ce3bc43cf8983def8601a8014 P 0bf3c1db9f574312be26a2ee04550de6: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.004s
I20260924 03:26:27.485409 28498 raft_consensus.cc:348] T 69172f8ce3bc43cf8983def8601a8014 P 15348a4c611047f39df72f27d07d37d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15348a4c611047f39df72f27d07d37d3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39927 } } peers { permanent_uuid: "993551e384ea4d85832e5a3affa6fec9" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 37377 } } peers { permanent_uuid: "0bf3c1db9f574312be26a2ee04550de6" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44577 } }
I20260924 03:26:27.485809 28498 raft_consensus.cc:374] T 69172f8ce3bc43cf8983def8601a8014 P 15348a4c611047f39df72f27d07d37d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:27.485903 28498 raft_consensus.cc:729] T 69172f8ce3bc43cf8983def8601a8014 P 15348a4c611047f39df72f27d07d37d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15348a4c611047f39df72f27d07d37d3, State: Initialized, Role: FOLLOWER
I20260924 03:26:27.486511 28498 consensus_queue.cc:260] T 69172f8ce3bc43cf8983def8601a8014 P 15348a4c611047f39df72f27d07d37d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15348a4c611047f39df72f27d07d37d3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39927 } } peers { permanent_uuid: "993551e384ea4d85832e5a3affa6fec9" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 37377 } } peers { permanent_uuid: "0bf3c1db9f574312be26a2ee04550de6" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44577 } }
I20260924 03:26:27.487587 28480 heartbeater.cc:503] Master 127.10.12.190:34759 was elected leader, sending a full tablet report...
I20260924 03:26:27.488307 28498 ts_tablet_manager.cc:1428] T 69172f8ce3bc43cf8983def8601a8014 P 15348a4c611047f39df72f27d07d37d3: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260924 03:26:27.491171 28500 tablet_bootstrap.cc:492] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9: No bootstrap required, opened a new log
I20260924 03:26:27.491472 28500 ts_tablet_manager.cc:1397] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.006s
I20260924 03:26:27.495034 28499 raft_consensus.cc:348] T 69172f8ce3bc43cf8983def8601a8014 P 0bf3c1db9f574312be26a2ee04550de6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15348a4c611047f39df72f27d07d37d3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39927 } } peers { permanent_uuid: "993551e384ea4d85832e5a3affa6fec9" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 37377 } } peers { permanent_uuid: "0bf3c1db9f574312be26a2ee04550de6" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44577 } }
I20260924 03:26:27.495410 28499 raft_consensus.cc:374] T 69172f8ce3bc43cf8983def8601a8014 P 0bf3c1db9f574312be26a2ee04550de6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:27.495503 28499 raft_consensus.cc:729] T 69172f8ce3bc43cf8983def8601a8014 P 0bf3c1db9f574312be26a2ee04550de6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bf3c1db9f574312be26a2ee04550de6, State: Initialized, Role: FOLLOWER
I20260924 03:26:27.496124 28499 consensus_queue.cc:260] T 69172f8ce3bc43cf8983def8601a8014 P 0bf3c1db9f574312be26a2ee04550de6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15348a4c611047f39df72f27d07d37d3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39927 } } peers { permanent_uuid: "993551e384ea4d85832e5a3affa6fec9" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 37377 } } peers { permanent_uuid: "0bf3c1db9f574312be26a2ee04550de6" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44577 } }
I20260924 03:26:27.497221 28350 heartbeater.cc:503] Master 127.10.12.190:34759 was elected leader, sending a full tablet report...
I20260924 03:26:27.497731 28499 ts_tablet_manager.cc:1428] T 69172f8ce3bc43cf8983def8601a8014 P 0bf3c1db9f574312be26a2ee04550de6: Time spent starting tablet: real 0.012s user 0.005s sys 0.008s
I20260924 03:26:27.500324 28500 raft_consensus.cc:348] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15348a4c611047f39df72f27d07d37d3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39927 } } peers { permanent_uuid: "993551e384ea4d85832e5a3affa6fec9" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 37377 } } peers { permanent_uuid: "0bf3c1db9f574312be26a2ee04550de6" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44577 } }
I20260924 03:26:27.500814 28500 raft_consensus.cc:374] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:27.500972 28500 raft_consensus.cc:729] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993551e384ea4d85832e5a3affa6fec9, State: Initialized, Role: FOLLOWER
I20260924 03:26:27.501591 28500 consensus_queue.cc:260] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15348a4c611047f39df72f27d07d37d3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39927 } } peers { permanent_uuid: "993551e384ea4d85832e5a3affa6fec9" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 37377 } } peers { permanent_uuid: "0bf3c1db9f574312be26a2ee04550de6" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44577 } }
I20260924 03:26:27.503286 28500 ts_tablet_manager.cc:1428] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
W20260924 03:26:27.608721 28481 tablet.cc:2369] T 69172f8ce3bc43cf8983def8601a8014 P 15348a4c611047f39df72f27d07d37d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:26:27.620330 28506 raft_consensus.cc:482] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:26:27.620674 28506 raft_consensus.cc:504] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15348a4c611047f39df72f27d07d37d3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39927 } } peers { permanent_uuid: "993551e384ea4d85832e5a3affa6fec9" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 37377 } } peers { permanent_uuid: "0bf3c1db9f574312be26a2ee04550de6" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44577 } }
I20260924 03:26:27.622498 28506 leader_election.cc:288] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15348a4c611047f39df72f27d07d37d3 (127.10.12.131:39927), 0bf3c1db9f574312be26a2ee04550de6 (127.10.12.130:44577)
I20260924 03:26:27.629715 28493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:27.622884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:43955 (local address 127.10.12.131:39927)
0924 03:26:27.623129 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:27.623137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:27.623728 (+ 591us) server_negotiation.cc:407] Connection header received
0924 03:26:27.624014 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:27.624020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:27.624110 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:27.624204 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:27.624820 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.625729 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.627802 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.628719 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.628853 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:27.628919 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:27.629339 (+ 420us) server_negotiation.cc:299] Negotiation successful
0924 03:26:27.629470 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 03:26:27.629729 28495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:27.623373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:40781 (local address 127.10.12.130:44577)
0924 03:26:27.623550 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:27.623554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:27.624109 (+ 555us) server_negotiation.cc:407] Connection header received
0924 03:26:27.624273 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:27.624278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:27.624439 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:27.624544 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:27.625149 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.626077 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.628062 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.628857 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.629018 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:27.629088 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:27.629351 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 03:26:27.629472 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 03:26:27.629830 28508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:27.622792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:44577 (local address 127.10.12.129:40781)
0924 03:26:27.624003 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:27.624020 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:27.624152 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:27.624577 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:27.624584 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:27.624606 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:27.624964 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:27.624976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.626242 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.626246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:27.627781 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:27.627796 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.629053 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.629058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:27.629198 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:27.629222 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 03:26:27.629305 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 03:26:27.629448 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1076,"mutex_wait_us":29,"thread_start_us":833,"threads_started":1}
I20260924 03:26:27.629881 28507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:27.622787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:39927 (local address 127.10.12.129:43955)
0924 03:26:27.623568 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:27.623591 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:27.623791 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:27.624235 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:27.624239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:27.624277 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:27.624673 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:27.624690 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.625852 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.625858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:27.627582 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:27.627597 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.628929 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.628938 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:27.629139 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:27.629204 (+ 65us) client_negotiation.cc:769] Sending connection context
0924 03:26:27.629282 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 03:26:27.629429 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":47,"thread_start_us":350,"threads_started":1}
I20260924 03:26:27.630985 28305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69172f8ce3bc43cf8983def8601a8014" candidate_uuid: "993551e384ea4d85832e5a3affa6fec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bf3c1db9f574312be26a2ee04550de6" is_pre_election: true
I20260924 03:26:27.631004 28435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69172f8ce3bc43cf8983def8601a8014" candidate_uuid: "993551e384ea4d85832e5a3affa6fec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15348a4c611047f39df72f27d07d37d3" is_pre_election: true
I20260924 03:26:27.631620 28435 raft_consensus.cc:2454] T 69172f8ce3bc43cf8983def8601a8014 P 15348a4c611047f39df72f27d07d37d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993551e384ea4d85832e5a3affa6fec9 in term 0.
I20260924 03:26:27.631621 28305 raft_consensus.cc:2454] T 69172f8ce3bc43cf8983def8601a8014 P 0bf3c1db9f574312be26a2ee04550de6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993551e384ea4d85832e5a3affa6fec9 in term 0.
I20260924 03:26:27.632405 28109 leader_election.cc:302] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bf3c1db9f574312be26a2ee04550de6, 993551e384ea4d85832e5a3affa6fec9; no voters:
I20260924 03:26:27.632854 28506 raft_consensus.cc:2781] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:26:27.633024 28506 raft_consensus.cc:482] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:26:27.633126 28506 raft_consensus.cc:3037] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:27.635502 28506 raft_consensus.cc:504] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15348a4c611047f39df72f27d07d37d3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39927 } } peers { permanent_uuid: "993551e384ea4d85832e5a3affa6fec9" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 37377 } } peers { permanent_uuid: "0bf3c1db9f574312be26a2ee04550de6" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44577 } }
I20260924 03:26:27.636211 28506 leader_election.cc:288] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9 [CANDIDATE]: Term 1 election: Requested vote from peers 15348a4c611047f39df72f27d07d37d3 (127.10.12.131:39927), 0bf3c1db9f574312be26a2ee04550de6 (127.10.12.130:44577)
I20260924 03:26:27.636720 28305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69172f8ce3bc43cf8983def8601a8014" candidate_uuid: "993551e384ea4d85832e5a3affa6fec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bf3c1db9f574312be26a2ee04550de6"
I20260924 03:26:27.636719 28435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69172f8ce3bc43cf8983def8601a8014" candidate_uuid: "993551e384ea4d85832e5a3affa6fec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15348a4c611047f39df72f27d07d37d3"
I20260924 03:26:27.636912 28305 raft_consensus.cc:3037] T 69172f8ce3bc43cf8983def8601a8014 P 0bf3c1db9f574312be26a2ee04550de6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:27.636958 28435 raft_consensus.cc:3037] T 69172f8ce3bc43cf8983def8601a8014 P 15348a4c611047f39df72f27d07d37d3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:27.639166 28435 raft_consensus.cc:2454] T 69172f8ce3bc43cf8983def8601a8014 P 15348a4c611047f39df72f27d07d37d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993551e384ea4d85832e5a3affa6fec9 in term 1.
I20260924 03:26:27.639266 28305 raft_consensus.cc:2454] T 69172f8ce3bc43cf8983def8601a8014 P 0bf3c1db9f574312be26a2ee04550de6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993551e384ea4d85832e5a3affa6fec9 in term 1.
I20260924 03:26:27.639631 28107 leader_election.cc:302] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15348a4c611047f39df72f27d07d37d3, 993551e384ea4d85832e5a3affa6fec9; no voters:
I20260924 03:26:27.640041 28506 raft_consensus.cc:2781] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9 [term 1 FOLLOWER]: Leader election won for term 1
W20260924 03:26:27.640444 28351 tablet.cc:2369] T 69172f8ce3bc43cf8983def8601a8014 P 0bf3c1db9f574312be26a2ee04550de6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:26:27.640600 28506 raft_consensus.cc:686] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9 [term 1 LEADER]: Becoming Leader. State: Replica: 993551e384ea4d85832e5a3affa6fec9, State: Running, Role: LEADER
I20260924 03:26:27.641273 28506 consensus_queue.cc:237] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15348a4c611047f39df72f27d07d37d3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39927 } } peers { permanent_uuid: "993551e384ea4d85832e5a3affa6fec9" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 37377 } } peers { permanent_uuid: "0bf3c1db9f574312be26a2ee04550de6" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44577 } }
I20260924 03:26:27.645892 28032 catalog_manager.cc:5495] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 993551e384ea4d85832e5a3affa6fec9 (127.10.12.129). New cstate: current_term: 1 leader_uuid: "993551e384ea4d85832e5a3affa6fec9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15348a4c611047f39df72f27d07d37d3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "993551e384ea4d85832e5a3affa6fec9" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bf3c1db9f574312be26a2ee04550de6" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44577 } health_report { overall_health: UNKNOWN } } }
W20260924 03:26:27.669797 28220 tablet.cc:2369] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:26:27.693591 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:27.688049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56130 (local address 127.10.12.129:37377)
0924 03:26:27.688254 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:27.688260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:27.688283 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:26:27.688424 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:27.688428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:27.688519 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:27.688651 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:26:27.689019 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.689719 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.691260 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.691698 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.691809 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:27.692477 (+ 668us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:26:27.692992 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:26:27.693126 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:27.693240 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 03:26:27.693327 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 03:26:27.703555 28435 raft_consensus.cc:1261] T 69172f8ce3bc43cf8983def8601a8014 P 15348a4c611047f39df72f27d07d37d3 [term 1 FOLLOWER]: Refusing update from remote peer 993551e384ea4d85832e5a3affa6fec9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:26:27.703586 28305 raft_consensus.cc:1261] T 69172f8ce3bc43cf8983def8601a8014 P 0bf3c1db9f574312be26a2ee04550de6 [term 1 FOLLOWER]: Refusing update from remote peer 993551e384ea4d85832e5a3affa6fec9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:26:27.704668 28509 consensus_queue.cc:1035] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15348a4c611047f39df72f27d07d37d3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 39927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:26:27.705171 28506 consensus_queue.cc:1035] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bf3c1db9f574312be26a2ee04550de6" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:26:27.719806 28514 mvcc.cc:205] Tried to move back new op lower bound from 7332742708022943744 to 7332742707782524928. Current Snapshot: MvccSnapshot[applied={T|T < 7332742708022943744}]
I20260924 03:26:27.734616 28032 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:26:27.734884 28032 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:26:27.738725 28032 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b14fdde53574424893dc719f31b8cff0: Sending DeleteTablet for 3 replicas of tablet 69172f8ce3bc43cf8983def8601a8014
I20260924 03:26:27.740296 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet 69172f8ce3bc43cf8983def8601a8014 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:26:27 UTC) from {username='slave'} at 127.0.0.1:56118
I20260924 03:26:27.740289 28415 tablet_service.cc:1514] Processing DeleteTablet for tablet 69172f8ce3bc43cf8983def8601a8014 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:26:27 UTC) from {username='slave'} at 127.0.0.1:42858
I20260924 03:26:27.740293 28285 tablet_service.cc:1514] Processing DeleteTablet for tablet 69172f8ce3bc43cf8983def8601a8014 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:26:27 UTC) from {username='slave'} at 127.0.0.1:34424
I20260924 03:26:27.742801 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 28092
I20260924 03:26:27.757884 28524 tablet_replica.cc:330] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9: stopping tablet replica
I20260924 03:26:27.758622 28524 raft_consensus.cc:2229] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:26:27.759516 28524 raft_consensus.cc:2258] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:26:27.760501 28523 tablet_replica.cc:330] T 69172f8ce3bc43cf8983def8601a8014 P 0bf3c1db9f574312be26a2ee04550de6: stopping tablet replica
I20260924 03:26:27.762706 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:27.744405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56132 (local address 127.10.12.129:37377)
0924 03:26:27.744614 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:27.744620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:27.749164 (+ 4544us) server_negotiation.cc:407] Connection header received
0924 03:26:27.750123 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:27.750129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:27.750263 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:27.750365 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:27.751596 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.753012 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.757916 (+ 4904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.758692 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.758812 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:27.760761 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:27.760772 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:27.760796 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:27.761703 (+ 907us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:27.761772 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:27.761800 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:27.761890 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:27.762195 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 03:26:27.762321 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 03:26:27.763839 28522 tablet_replica.cc:330] T 69172f8ce3bc43cf8983def8601a8014 P 15348a4c611047f39df72f27d07d37d3: stopping tablet replica
I20260924 03:26:27.764073 28214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:27.764071 28523 raft_consensus.cc:2229] T 69172f8ce3bc43cf8983def8601a8014 P 0bf3c1db9f574312be26a2ee04550de6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:26:27.764274 28522 raft_consensus.cc:2229] T 69172f8ce3bc43cf8983def8601a8014 P 15348a4c611047f39df72f27d07d37d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:26:27.764675 28523 raft_consensus.cc:2258] T 69172f8ce3bc43cf8983def8601a8014 P 0bf3c1db9f574312be26a2ee04550de6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:26:27.764791 28522 raft_consensus.cc:2258] T 69172f8ce3bc43cf8983def8601a8014 P 15348a4c611047f39df72f27d07d37d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:26:27.766876 28523 ts_tablet_manager.cc:1905] T 69172f8ce3bc43cf8983def8601a8014 P 0bf3c1db9f574312be26a2ee04550de6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:26:27.767040 28522 ts_tablet_manager.cc:1905] T 69172f8ce3bc43cf8983def8601a8014 P 15348a4c611047f39df72f27d07d37d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:26:27.773692 28523 ts_tablet_manager.cc:1918] T 69172f8ce3bc43cf8983def8601a8014 P 0bf3c1db9f574312be26a2ee04550de6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:26:27.773896 28523 log.cc:1196] T 69172f8ce3bc43cf8983def8601a8014 P 0bf3c1db9f574312be26a2ee04550de6: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/69172f8ce3bc43cf8983def8601a8014
I20260924 03:26:27.774433 28523 ts_tablet_manager.cc:1939] T 69172f8ce3bc43cf8983def8601a8014 P 0bf3c1db9f574312be26a2ee04550de6: Deleting consensus metadata
I20260924 03:26:27.774870 28522 ts_tablet_manager.cc:1918] T 69172f8ce3bc43cf8983def8601a8014 P 15348a4c611047f39df72f27d07d37d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:26:27.775028 28522 log.cc:1196] T 69172f8ce3bc43cf8983def8601a8014 P 15348a4c611047f39df72f27d07d37d3: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/69172f8ce3bc43cf8983def8601a8014
I20260924 03:26:27.775437 28522 ts_tablet_manager.cc:1939] T 69172f8ce3bc43cf8983def8601a8014 P 15348a4c611047f39df72f27d07d37d3: Deleting consensus metadata
I20260924 03:26:27.776619 28019 catalog_manager.cc:4841] TS 0bf3c1db9f574312be26a2ee04550de6 (127.10.12.130:44577): tablet 69172f8ce3bc43cf8983def8601a8014 (table test-table [id=b28bdedcc4ed49eea087004744c242b8]) successfully deleted
I20260924 03:26:27.776990 28017 catalog_manager.cc:4841] TS 15348a4c611047f39df72f27d07d37d3 (127.10.12.131:39927): tablet 69172f8ce3bc43cf8983def8601a8014 (table test-table [id=b28bdedcc4ed49eea087004744c242b8]) successfully deleted
I20260924 03:26:27.764330 28524 ts_tablet_manager.cc:1905] T 69172f8ce3bc43cf8983def8601a8014 P 993551e384ea4d85832e5a3affa6fec9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:26:27.866902 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 28092
W20260924 03:26:27.881747 28017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.12.129:37377 (error 108)
I20260924 03:26:27.882000 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 28223
I20260924 03:26:27.884116 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:27.883402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:37377 (local address 127.0.0.1:56140)
0924 03:26:27.883607 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:27.883870 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 03:26:27.884428 28017 catalog_manager.cc:4584] TS 993551e384ea4d85832e5a3affa6fec9 (127.10.12.129:37377): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 69172f8ce3bc43cf8983def8601a8014: Network error: Client connection negotiation failed: client connection to 127.10.12.129:37377: connect: Connection refused (error 111)
I20260924 03:26:27.890591 28495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:27.882734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34432 (local address 127.10.12.130:44577)
0924 03:26:27.882929 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:27.882935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:27.883029 (+ 94us) server_negotiation.cc:407] Connection header received
0924 03:26:27.883209 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:27.883213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:27.883332 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:27.883422 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:27.884105 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.884965 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.886409 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:27.887092 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:27.887274 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:27.888796 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:27.888808 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:27.888825 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:27.889618 (+ 793us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:27.889684 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:27.889711 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:27.889837 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:27.890072 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 03:26:27.890200 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 03:26:27.891466 28345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:27.938125 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:27.937625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:37377 (local address 127.0.0.1:56152)
0924 03:26:27.937874 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:27.937970 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 03:26:27.939235 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:27.938753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:37377 (local address 127.0.0.1:56162)
0924 03:26:27.938979 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:27.939072 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 03:26:27.984916 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 28223
I20260924 03:26:27.999289 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 28353
I20260924 03:26:28.000003 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:27.999449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:37377 (local address 127.0.0.1:56176)
0924 03:26:27.999663 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:27.999770 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 03:26:28.001238 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:28.000760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:37377 (local address 127.0.0.1:56184)
0924 03:26:28.000993 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:28.001085 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260924 03:26:28.007210 28493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:28.000090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42870 (local address 127.10.12.131:39927)
0924 03:26:28.000262 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:28.000268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:28.000289 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:26:28.000491 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:28.000496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:28.000601 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:28.000691 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:28.001192 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:28.002015 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:28.003693 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:28.004299 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:28.004426 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:28.005761 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:28.005770 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:28.005785 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:28.006448 (+ 663us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:28.006500 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:28.006528 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:28.006627 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:28.006834 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 03:26:28.006944 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:26:28.007860 28475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:28.081818 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:28.081324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:37377 (local address 127.0.0.1:56196)
0924 03:26:28.081554 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:28.081653 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 03:26:28.082991 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:28.082536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:37377 (local address 127.0.0.1:56202)
0924 03:26:28.082720 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:28.082818 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260924 03:26:28.098429 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 28353
I20260924 03:26:28.111811 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 28002
I20260924 03:26:28.112986 28063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:28.223147 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 28002
2026-09-24T03:26:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2946 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:26:28 dist-test-slave-00qs krb5kdc[28533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:26:28 dist-test-slave-00qs krb5kdc[28533](info): set up 2 sockets
Sep 24 03:26:28 dist-test-slave-00qs krb5kdc[28533](info): commencing operation
krb5kdc: starting...
W20260924 03:26:29.256227 10290 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:26:29 dist-test-slave-00qs krb5kdc[28533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:26:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:26:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:26:29.517611 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:32897
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:41701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:32897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:26:29.935559 28546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:29.936085 28546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:29.936244 28546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:29.946026 28546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:26:29.946249 28546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:29.946336 28546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:29.946431 28546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:26:29.946496 28546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:26:29.959101 28546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:41701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:32897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:32897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:29.961524 28546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:29.963534 28546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:29.980391 28551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:29.980275 28552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:29.980792 28554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:29.984004 28546 server_base.cc:1034] running on GCE node
I20260924 03:26:29.985095 28546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:29.987031 28546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:29.988282 28546 hybrid_clock.cc:648] HybridClock initialized: now 1790220389988201 us; error 78 us; skew 500 ppm
I20260924 03:26:29.988757 28546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:26:29 dist-test-slave-00qs krb5kdc[28533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220389, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:26:29.992818 28546 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:26:29.994853 28546 webserver.cc:460] Webserver started at http://127.10.12.190:33185/ using document root <none> and password file <none>
I20260924 03:26:29.995784 28546 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:29.995921 28546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:29.996327 28546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:30.000206 28546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "3566ae82ebde4f39b82b84eeca056dbe"
format_stamp: "Formatted at 2026-09-24 03:26:29 on dist-test-slave-00qs"
server_key: "3d7b15c08791330b0a4a07cfa2b70bb1"
server_key_iv: "75b5f947db9e82a19df3172c6f26ac5a"
server_key_version: "encryptionkey@0"
I20260924 03:26:30.001058 28546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "3566ae82ebde4f39b82b84eeca056dbe"
format_stamp: "Formatted at 2026-09-24 03:26:29 on dist-test-slave-00qs"
server_key: "3d7b15c08791330b0a4a07cfa2b70bb1"
server_key_iv: "75b5f947db9e82a19df3172c6f26ac5a"
server_key_version: "encryptionkey@0"
I20260924 03:26:30.008827 28546 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260924 03:26:30.013970 28561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:30.016120 28546 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.003s
I20260924 03:26:30.016353 28546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "3566ae82ebde4f39b82b84eeca056dbe"
format_stamp: "Formatted at 2026-09-24 03:26:29 on dist-test-slave-00qs"
server_key: "3d7b15c08791330b0a4a07cfa2b70bb1"
server_key_iv: "75b5f947db9e82a19df3172c6f26ac5a"
server_key_version: "encryptionkey@0"
I20260924 03:26:30.016608 28546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:30.055881 28546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:30.061663 28546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:30.062000 28546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:30.085021 28546 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:32897
I20260924 03:26:30.085021 28612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:32897 every 8 connection(s)
I20260924 03:26:30.086977 28546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:26:30.092365 28613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:30.094585 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 28546
I20260924 03:26:30.094805 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:26:30.095484 10290 external_mini_cluster.cc:1428] Setting key 17513feaadbb192120602de5889d219b
I20260924 03:26:30.106770 28613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3566ae82ebde4f39b82b84eeca056dbe: Bootstrap starting.
I20260924 03:26:30.111090 28613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3566ae82ebde4f39b82b84eeca056dbe: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:30.112550 28613 log.cc:824] T 00000000000000000000000000000000 P 3566ae82ebde4f39b82b84eeca056dbe: Log is configured to *not* fsync() on all Append() calls
Sep 24 03:26:30 dist-test-slave-00qs krb5kdc[28533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:26:30.116638 28613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3566ae82ebde4f39b82b84eeca056dbe: No bootstrap required, opened a new log
I20260924 03:26:30.124284 28616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:30.098137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49108 (local address 127.10.12.190:32897)
0924 03:26:30.099210 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:30.099225 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:30.099283 (+ 58us) server_negotiation.cc:407] Connection header received
0924 03:26:30.101491 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:30.101525 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:30.102073 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:30.102602 (+ 529us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:30.104435 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:30.105983 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:30.107820 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:30.108751 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:30.108984 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:30.114457 (+ 5473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:30.114468 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:30.114490 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:30.114535 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:30.118194 (+ 3659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:30.119358 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:30.119366 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:30.119379 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:30.119587 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:30.120156 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:30.120161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:30.120163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:30.120929 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:30.121292 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:30.121992 (+ 700us) server_negotiation.cc:299] Negotiation successful
0924 03:26:30.122791 (+ 799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":496,"threads_started":1}
I20260924 03:26:30.126900 28613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3566ae82ebde4f39b82b84eeca056dbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3566ae82ebde4f39b82b84eeca056dbe" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 32897 } }
I20260924 03:26:30.127388 28613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3566ae82ebde4f39b82b84eeca056dbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:30.127521 28613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3566ae82ebde4f39b82b84eeca056dbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3566ae82ebde4f39b82b84eeca056dbe, State: Initialized, Role: FOLLOWER
I20260924 03:26:30.128283 28613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3566ae82ebde4f39b82b84eeca056dbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3566ae82ebde4f39b82b84eeca056dbe" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 32897 } }
I20260924 03:26:30.128545 28613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3566ae82ebde4f39b82b84eeca056dbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:26:30.128654 28613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3566ae82ebde4f39b82b84eeca056dbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:26:30.128803 28613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3566ae82ebde4f39b82b84eeca056dbe [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:30.131577 28613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3566ae82ebde4f39b82b84eeca056dbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3566ae82ebde4f39b82b84eeca056dbe" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 32897 } }
I20260924 03:26:30.132345 28613 leader_election.cc:302] T 00000000000000000000000000000000 P 3566ae82ebde4f39b82b84eeca056dbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3566ae82ebde4f39b82b84eeca056dbe; no voters:
I20260924 03:26:30.133206 28613 leader_election.cc:288] T 00000000000000000000000000000000 P 3566ae82ebde4f39b82b84eeca056dbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:26:30.133374 28618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3566ae82ebde4f39b82b84eeca056dbe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:26:30.134495 28618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3566ae82ebde4f39b82b84eeca056dbe [term 1 LEADER]: Becoming Leader. State: Replica: 3566ae82ebde4f39b82b84eeca056dbe, State: Running, Role: LEADER
I20260924 03:26:30.135119 28618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3566ae82ebde4f39b82b84eeca056dbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3566ae82ebde4f39b82b84eeca056dbe" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 32897 } }
I20260924 03:26:30.135831 28613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3566ae82ebde4f39b82b84eeca056dbe [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:26:30.138736 28619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3566ae82ebde4f39b82b84eeca056dbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3566ae82ebde4f39b82b84eeca056dbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3566ae82ebde4f39b82b84eeca056dbe" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 32897 } } }
I20260924 03:26:30.139195 28619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3566ae82ebde4f39b82b84eeca056dbe [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:30.140040 28620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3566ae82ebde4f39b82b84eeca056dbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3566ae82ebde4f39b82b84eeca056dbe. Latest consensus state: current_term: 1 leader_uuid: "3566ae82ebde4f39b82b84eeca056dbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3566ae82ebde4f39b82b84eeca056dbe" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 32897 } } }
I20260924 03:26:30.140403 28620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3566ae82ebde4f39b82b84eeca056dbe [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:30.140301 28623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:26:30.151070 28623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:26:30.164947 28623 catalog_manager.cc:1347] Generated new cluster ID: d26a7665edb547b29de17f5be162a9bb
I20260924 03:26:30.165124 28623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:26:30.184962 28623 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:26:30.186782 28623 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:26:30.195312 28623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3566ae82ebde4f39b82b84eeca056dbe: Generated new TSK 0
I20260924 03:26:30.197218 28623 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
I20260924 03:26:30.281827 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:32897
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:41701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:26:30.689033 28639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:30.689354 28639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:30.689460 28639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:30.697474 28639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:30.697619 28639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:30.697768 28639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:26:30.708357 28639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:41701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:32897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:30.709800 28639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:30.711258 28639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:30.725589 28644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:30.725561 28645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:30.725636 28647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:30.726303 28639 server_base.cc:1034] running on GCE node
I20260924 03:26:30.727023 28639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:30.728226 28639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:30.729519 28639 hybrid_clock.cc:648] HybridClock initialized: now 1790220390729480 us; error 35 us; skew 500 ppm
I20260924 03:26:30.729868 28639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:26:30 dist-test-slave-00qs krb5kdc[28533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220390, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:26:30.733409 28639 init.cc:377] Logged in from keytab as kudu/127.10.12.129@KRBTEST.COM (short username kudu)
I20260924 03:26:30.735220 28639 webserver.cc:460] Webserver started at http://127.10.12.129:39943/ using document root <none> and password file <none>
I20260924 03:26:30.736095 28639 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:30.736239 28639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:30.736613 28639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:30.739882 28639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "7f4f6d95f2c14d5382a43b0fdedfee01"
format_stamp: "Formatted at 2026-09-24 03:26:30 on dist-test-slave-00qs"
server_key: "2cea461e9b4ac2793e964e2b3bd6d886"
server_key_iv: "22aae22d4ce7bf5bdc62987b3f9cd6e4"
server_key_version: "encryptionkey@0"
I20260924 03:26:30.740727 28639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "7f4f6d95f2c14d5382a43b0fdedfee01"
format_stamp: "Formatted at 2026-09-24 03:26:30 on dist-test-slave-00qs"
server_key: "2cea461e9b4ac2793e964e2b3bd6d886"
server_key_iv: "22aae22d4ce7bf5bdc62987b3f9cd6e4"
server_key_version: "encryptionkey@0"
I20260924 03:26:30.746483 28639 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:26:30.750612 28654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:30.752691 28639 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:26:30.752915 28639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "7f4f6d95f2c14d5382a43b0fdedfee01"
format_stamp: "Formatted at 2026-09-24 03:26:30 on dist-test-slave-00qs"
server_key: "2cea461e9b4ac2793e964e2b3bd6d886"
server_key_iv: "22aae22d4ce7bf5bdc62987b3f9cd6e4"
server_key_version: "encryptionkey@0"
I20260924 03:26:30.753129 28639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:30.778867 28639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:30.783629 28639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:30.783918 28639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:30.784992 28639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:26:30.786502 28639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:26:30.786583 28639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:30.786702 28639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:26:30.786757 28639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:30.819389 28639 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:34273
I20260924 03:26:30.819490 28766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:34273 every 8 connection(s)
I20260924 03:26:30.820925 28639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:26:30.824469 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 28639
I20260924 03:26:30.824656 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:26:30.825138 10290 external_mini_cluster.cc:1428] Setting key 06c06c34b160e85314bc640111fcf2ac
Sep 24 03:26:30 dist-test-slave-00qs krb5kdc[28533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220390, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:26:30.844944 28769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:30.823757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:40563 (local address 127.10.12.190:32897)
0924 03:26:30.824421 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:30.824428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:30.825601 (+ 1173us) server_negotiation.cc:407] Connection header received
0924 03:26:30.827543 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:30.827549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:30.827638 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:30.827757 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:30.830041 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:30.831185 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:30.833086 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:30.833688 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:30.833823 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:30.838359 (+ 4536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:30.838364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:30.838368 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:30.838460 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:30.841186 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:30.842230 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:30.842236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:30.842238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:30.842389 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:30.842980 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:30.842985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:30.842987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:30.843283 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:30.843458 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:30.844437 (+ 979us) server_negotiation.cc:299] Negotiation successful
0924 03:26:30.844648 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":375,"threads_started":1}
I20260924 03:26:30.846273 28770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:30.824504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:32897 (local address 127.10.12.129:40563)
0924 03:26:30.825393 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:30.825461 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:30.827236 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:30.828074 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:30.828084 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:30.829140 (+ 1056us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:26:30.829801 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:30.829822 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:30.831468 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:30.831473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:30.832904 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:30.832921 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:30.833987 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:30.833993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:30.834222 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:30.835137 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:26:30.835163 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:26:30.838024 (+ 2861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:26:30.841389 (+ 3365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:26:30.841399 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:26:30.841420 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:26:30.842051 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:26:30.842556 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:26:30.842561 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:26:30.842565 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:26:30.842842 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:26:30.843407 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:26:30.843417 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:26:30.843880 (+ 463us) client_negotiation.cc:769] Sending connection context
0924 03:26:30.844247 (+ 367us) client_negotiation.cc:240] Negotiation successful
0924 03:26:30.844826 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":345,"threads_started":1}
I20260924 03:26:30.848541 28767 heartbeater.cc:347] Connected to a master server at 127.10.12.190:32897
I20260924 03:26:30.848904 28767 heartbeater.cc:464] Registering TS with master...
I20260924 03:26:30.849749 28767 heartbeater.cc:511] Master 127.10.12.190:32897 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.12.130@KRBTEST.COM; defaulting to no policy
I20260924 03:26:30.852500 28578 ts_manager.cc:194] Registered new tserver with Master: 7f4f6d95f2c14d5382a43b0fdedfee01 (127.10.12.129:34273)
I20260924 03:26:30.854190 28578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.129@KRBTEST.COM'} at 127.10.12.129:40563
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
I20260924 03:26:30.887851 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:32897
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:41701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:26:31.240386 28773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:31.240738 28773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:31.240880 28773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:31.248385 28773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:31.248518 28773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:31.248670 28773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:26:31.259812 28773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:41701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:32897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:31.261198 28773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:31.262526 28773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:31.276594 28778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:31.276561 28781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:31.276543 28779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:31.277278 28773 server_base.cc:1034] running on GCE node
I20260924 03:26:31.277976 28773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:31.279184 28773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:31.280435 28773 hybrid_clock.cc:648] HybridClock initialized: now 1790220391280354 us; error 63 us; skew 500 ppm
I20260924 03:26:31.280786 28773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:26:31 dist-test-slave-00qs krb5kdc[28533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220391, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:26:31.284195 28773 init.cc:377] Logged in from keytab as kudu/127.10.12.130@KRBTEST.COM (short username kudu)
I20260924 03:26:31.285954 28773 webserver.cc:460] Webserver started at http://127.10.12.130:34637/ using document root <none> and password file <none>
I20260924 03:26:31.286686 28773 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:31.286821 28773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:31.287132 28773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:31.290213 28773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "5934bf6602ee4aa681d17199b28d2f39"
format_stamp: "Formatted at 2026-09-24 03:26:31 on dist-test-slave-00qs"
server_key: "03652836648e314865854e595fc7e633"
server_key_iv: "ce1aefb901bdf82bc6b44c374ad8d331"
server_key_version: "encryptionkey@0"
I20260924 03:26:31.291057 28773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "5934bf6602ee4aa681d17199b28d2f39"
format_stamp: "Formatted at 2026-09-24 03:26:31 on dist-test-slave-00qs"
server_key: "03652836648e314865854e595fc7e633"
server_key_iv: "ce1aefb901bdf82bc6b44c374ad8d331"
server_key_version: "encryptionkey@0"
I20260924 03:26:31.297425 28773 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 03:26:31.301565 28788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:31.303387 28773 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:26:31.303596 28773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "5934bf6602ee4aa681d17199b28d2f39"
format_stamp: "Formatted at 2026-09-24 03:26:31 on dist-test-slave-00qs"
server_key: "03652836648e314865854e595fc7e633"
server_key_iv: "ce1aefb901bdf82bc6b44c374ad8d331"
server_key_version: "encryptionkey@0"
I20260924 03:26:31.303812 28773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:31.316404 28773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:31.320799 28773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:31.321101 28773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:31.322144 28773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:26:31.323558 28773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:26:31.323638 28773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:31.323752 28773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:26:31.323802 28773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:31.356887 28773 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:45179
I20260924 03:26:31.356922 28900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:45179 every 8 connection(s)
I20260924 03:26:31.358348 28773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:26:31.363524 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 28773
I20260924 03:26:31.363688 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:26:31.364164 10290 external_mini_cluster.cc:1428] Setting key 294f021c4ea41b624faf647375edcc19
Sep 24 03:26:31 dist-test-slave-00qs krb5kdc[28533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220391, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:26:31.379110 28903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:31.361007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:51407 (local address 127.10.12.190:32897)
0924 03:26:31.361496 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:31.361501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:31.362451 (+ 950us) server_negotiation.cc:407] Connection header received
0924 03:26:31.364396 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:31.364403 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:31.364499 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:31.364603 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:31.366369 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:31.367115 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:31.368743 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:31.369206 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:31.369319 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:31.372808 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:31.372813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:31.372817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:31.372853 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:31.375845 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:31.376686 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:31.376691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:31.376693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:31.376829 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:31.377271 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:31.377274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:31.377277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:31.377575 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:31.377715 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:31.378469 (+ 754us) server_negotiation.cc:299] Negotiation successful
0924 03:26:31.378721 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":219,"threads_started":1}
I20260924 03:26:31.380172 28904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:31.361566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:32897 (local address 127.10.12.130:51407)
0924 03:26:31.362260 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:31.362296 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:31.364100 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:31.364871 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:31.364884 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:31.365511 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:26:31.366194 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:31.366215 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:31.367264 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:31.367268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:31.368583 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:31.368596 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:31.369442 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:31.369450 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:31.369672 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:31.370329 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:26:31.370353 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:26:31.372548 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:26:31.376043 (+ 3495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:26:31.376054 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:26:31.376071 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:26:31.376545 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:26:31.376941 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:26:31.376945 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:26:31.376949 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:26:31.377148 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:26:31.377712 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:26:31.377721 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:26:31.378082 (+ 361us) client_negotiation.cc:769] Sending connection context
0924 03:26:31.378342 (+ 260us) client_negotiation.cc:240] Negotiation successful
0924 03:26:31.378914 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":264,"threads_started":1}
I20260924 03:26:31.381723 28901 heartbeater.cc:347] Connected to a master server at 127.10.12.190:32897
I20260924 03:26:31.382125 28901 heartbeater.cc:464] Registering TS with master...
I20260924 03:26:31.383090 28901 heartbeater.cc:511] Master 127.10.12.190:32897 requested a full tablet report, sending...
I20260924 03:26:31.384936 28578 ts_manager.cc:194] Registered new tserver with Master: 5934bf6602ee4aa681d17199b28d2f39 (127.10.12.130:45179)
I20260924 03:26:31.385762 28578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.130@KRBTEST.COM'} at 127.10.12.130:51407
WARNING: no policy specified for kudu/127.10.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
I20260924 03:26:31.420531 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:32897
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:41701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:26:31.784889 28907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:31.785256 28907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:31.785357 28907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:31.794489 28907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:31.794651 28907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:31.794770 28907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:26:31.805287 28907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:41701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:32897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:31.807775 28907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:31.809464 28907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:31.824286 28912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:31.824256 28913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:31.824872 28915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:31.825605 28907 server_base.cc:1034] running on GCE node
I20260924 03:26:31.826390 28907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:31.827489 28907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:31.828724 28907 hybrid_clock.cc:648] HybridClock initialized: now 1790220391828679 us; error 37 us; skew 500 ppm
I20260924 03:26:31.829072 28907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:26:31 dist-test-slave-00qs krb5kdc[28533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220391, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:26:31.832536 28907 init.cc:377] Logged in from keytab as kudu/127.10.12.131@KRBTEST.COM (short username kudu)
I20260924 03:26:31.834368 28907 webserver.cc:460] Webserver started at http://127.10.12.131:35997/ using document root <none> and password file <none>
I20260924 03:26:31.835120 28907 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:31.835249 28907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:31.835568 28907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:31.838868 28907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "13b3aefe667449adab5e6821e840a455"
format_stamp: "Formatted at 2026-09-24 03:26:31 on dist-test-slave-00qs"
server_key: "eeb95f5704b625b21c72cf5fba1ccc0c"
server_key_iv: "cb46f00fb8a6b21c39a5a8cba2d25f0a"
server_key_version: "encryptionkey@0"
I20260924 03:26:31.839700 28907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "13b3aefe667449adab5e6821e840a455"
format_stamp: "Formatted at 2026-09-24 03:26:31 on dist-test-slave-00qs"
server_key: "eeb95f5704b625b21c72cf5fba1ccc0c"
server_key_iv: "cb46f00fb8a6b21c39a5a8cba2d25f0a"
server_key_version: "encryptionkey@0"
I20260924 03:26:31.845558 28907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 03:26:31.850157 28922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:31.851894 28907 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 03:26:31.852103 28907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "13b3aefe667449adab5e6821e840a455"
format_stamp: "Formatted at 2026-09-24 03:26:31 on dist-test-slave-00qs"
server_key: "eeb95f5704b625b21c72cf5fba1ccc0c"
server_key_iv: "cb46f00fb8a6b21c39a5a8cba2d25f0a"
server_key_version: "encryptionkey@0"
I20260924 03:26:31.852319 28907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:31.858031 28767 heartbeater.cc:503] Master 127.10.12.190:32897 was elected leader, sending a full tablet report...
I20260924 03:26:31.868956 28907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:31.874737 28907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:31.875061 28907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:31.876379 28907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:26:31.878939 28907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:26:31.879071 28907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:31.879225 28907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:26:31.879556 28907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:31.918633 28907 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:45853
I20260924 03:26:31.918692 29034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:45853 every 8 connection(s)
I20260924 03:26:31.920001 28907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:26:31.928387 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 28907
I20260924 03:26:31.928586 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:26:31.928996 10290 external_mini_cluster.cc:1428] Setting key c493757d2e9c0f983658e5759036e626
Sep 24 03:26:31 dist-test-slave-00qs krb5kdc[28533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220391, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:26:31.944278 29037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:31.922902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:59707 (local address 127.10.12.190:32897)
0924 03:26:31.923491 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:31.923496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:31.924492 (+ 996us) server_negotiation.cc:407] Connection header received
0924 03:26:31.927195 (+ 2703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:31.927201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:31.927313 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:31.927431 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:31.929337 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:31.930276 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:31.932143 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:31.932629 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:31.932737 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:31.936480 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:31.936488 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:31.936493 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:31.936569 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:31.939623 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:31.941157 (+ 1534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:31.941162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:31.941165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:31.941325 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:31.941982 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:31.941988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:31.941991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:31.942368 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:31.942556 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:31.943542 (+ 986us) server_negotiation.cc:299] Negotiation successful
0924 03:26:31.943811 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":277,"threads_started":1}
I20260924 03:26:31.945266 29038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:31.923515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:32897 (local address 127.10.12.131:59707)
0924 03:26:31.924325 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:31.924372 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:31.926600 (+ 2228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:31.927768 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:31.927779 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:31.928452 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:26:31.929142 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:31.929166 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:31.930499 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:31.930504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:31.932008 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:31.932026 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:31.932842 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:31.932848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:31.933087 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:31.933844 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:26:31.933868 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:26:31.936192 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:26:31.939946 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:26:31.940015 (+ 69us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:26:31.940044 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:26:31.940960 (+ 916us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:26:31.941538 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:26:31.941545 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:26:31.941548 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:26:31.941816 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:26:31.942544 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:26:31.942552 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:26:31.943041 (+ 489us) client_negotiation.cc:769] Sending connection context
0924 03:26:31.943374 (+ 333us) client_negotiation.cc:240] Negotiation successful
0924 03:26:31.943861 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":380,"threads_started":1}
I20260924 03:26:31.946734 29035 heartbeater.cc:347] Connected to a master server at 127.10.12.190:32897
I20260924 03:26:31.947079 29035 heartbeater.cc:464] Registering TS with master...
I20260924 03:26:31.947801 29035 heartbeater.cc:511] Master 127.10.12.190:32897 requested a full tablet report, sending...
I20260924 03:26:31.949514 28578 ts_manager.cc:194] Registered new tserver with Master: 13b3aefe667449adab5e6821e840a455 (127.10.12.131:45853)
I20260924 03:26:31.950456 28578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.131@KRBTEST.COM'} at 127.10.12.131:59707
I20260924 03:26:31.957903 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:26:31.979290 29037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:31.962730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49120 (local address 127.10.12.190:32897)
0924 03:26:31.962927 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:31.962932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:31.963256 (+ 324us) server_negotiation.cc:407] Connection header received
0924 03:26:31.963383 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:31.963388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:31.963470 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:31.963562 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:31.965163 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:31.966275 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:31.968263 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:31.968936 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:31.969054 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:31.972082 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:31.972088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:31.972091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:31.972132 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:31.975284 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:31.976539 (+ 1255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:31.976545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:31.976549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:31.976720 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:31.977474 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:31.977480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:31.977484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:31.977993 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:31.978281 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:31.978745 (+ 464us) server_negotiation.cc:299] Negotiation successful
0924 03:26:31.978960 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 03:26:31.984803 28578 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:26:31.992028 28578 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:26:32.029731 29051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:32.007502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45828 (local address 127.10.12.130:45179)
0924 03:26:32.019099 (+ 11597us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:32.019109 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:32.020328 (+ 1219us) server_negotiation.cc:407] Connection header received
0924 03:26:32.020429 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:32.020436 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:32.020651 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:32.020882 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:32.021985 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.023557 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.026741 (+ 3184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.028343 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.028485 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:32.028654 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:32.029201 (+ 547us) server_negotiation.cc:299] Negotiation successful
0924 03:26:32.029425 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11396,"thread_start_us":11283,"threads_started":1}
I20260924 03:26:32.029801 29048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:32.007213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:45179 (local address 127.0.0.1:45828)
0924 03:26:32.020024 (+ 12811us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:32.020044 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:32.020173 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:32.021126 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:32.021130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:32.021160 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:32.021730 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:32.021747 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.023704 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.023709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:32.026552 (+ 2843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:32.026570 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.028769 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.028774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:32.028954 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:32.028994 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 03:26:32.029120 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 03:26:32.029342 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12594,"mutex_wait_us":36,"thread_start_us":11923,"threads_started":1}
I20260924 03:26:32.030967 29050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:32.006643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:45853 (local address 127.0.0.1:44670)
0924 03:26:32.018037 (+ 11394us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:32.018098 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:32.018369 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:32.020932 (+ 2563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:32.020941 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:32.020993 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:32.021631 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:32.021646 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.023925 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.023931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:32.027760 (+ 3829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:32.027776 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.030020 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.030027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:32.030237 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:32.030272 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 03:26:32.030410 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 03:26:32.030551 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11033,"mutex_wait_us":3,"thread_start_us":12572,"threads_started":1}
I20260924 03:26:32.030998 29049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:32.006928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44670 (local address 127.10.12.131:45853)
0924 03:26:32.019781 (+ 12853us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:32.019789 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:32.019827 (+ 38us) server_negotiation.cc:407] Connection header received
0924 03:26:32.019935 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:32.019944 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:32.020375 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:32.020577 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:32.021881 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.023690 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.027986 (+ 4296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.029477 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.029577 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:32.029804 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:32.030477 (+ 673us) server_negotiation.cc:299] Negotiation successful
0924 03:26:32.030660 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12683,"thread_start_us":12507,"threads_started":1}
I20260924 03:26:32.034564 29053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:32.021255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:34273 (local address 127.0.0.1:44092)
0924 03:26:32.026083 (+ 4828us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:32.026100 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:32.026245 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:32.026882 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:32.026895 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:32.026923 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:32.027452 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:32.027469 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.029507 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.029512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:32.031767 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:32.031786 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.033702 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.033708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:32.033884 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:32.033919 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 03:26:32.034015 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 03:26:32.034162 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4703,"mutex_wait_us":1,"thread_start_us":4554,"threads_started":1}
I20260924 03:26:32.034562 29052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:32.020461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44092 (local address 127.10.12.129:34273)
0924 03:26:32.025682 (+ 5221us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:32.025688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:32.026219 (+ 531us) server_negotiation.cc:407] Connection header received
0924 03:26:32.026417 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:32.026423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:32.026665 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:32.026880 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:32.027696 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.029290 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.032054 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.033475 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.033622 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:32.033858 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:32.034077 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 03:26:32.034246 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5052,"thread_start_us":4859,"threads_started":1}
I20260924 03:26:32.034709 28836 tablet_service.cc:1467] Processing CreateTablet for tablet 237c3af7733947e685cc90f998da736e (DEFAULT_TABLE table=test-table [id=f1474885976b49af8e4628b5d139f709]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:26:32.036616 28836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 237c3af7733947e685cc90f998da736e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:32.036073 28970 tablet_service.cc:1467] Processing CreateTablet for tablet 237c3af7733947e685cc90f998da736e (DEFAULT_TABLE table=test-table [id=f1474885976b49af8e4628b5d139f709]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:26:32.037519 28970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 237c3af7733947e685cc90f998da736e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:32.037813 28702 tablet_service.cc:1467] Processing CreateTablet for tablet 237c3af7733947e685cc90f998da736e (DEFAULT_TABLE table=test-table [id=f1474885976b49af8e4628b5d139f709]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:26:32.038967 28702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 237c3af7733947e685cc90f998da736e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:32.052037 29054 tablet_bootstrap.cc:492] T 237c3af7733947e685cc90f998da736e P 7f4f6d95f2c14d5382a43b0fdedfee01: Bootstrap starting.
I20260924 03:26:32.056646 29054 tablet_bootstrap.cc:654] T 237c3af7733947e685cc90f998da736e P 7f4f6d95f2c14d5382a43b0fdedfee01: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:32.058054 29054 log.cc:824] T 237c3af7733947e685cc90f998da736e P 7f4f6d95f2c14d5382a43b0fdedfee01: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:32.058158 29055 tablet_bootstrap.cc:492] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455: Bootstrap starting.
I20260924 03:26:32.058161 29056 tablet_bootstrap.cc:492] T 237c3af7733947e685cc90f998da736e P 5934bf6602ee4aa681d17199b28d2f39: Bootstrap starting.
I20260924 03:26:32.061748 29054 tablet_bootstrap.cc:492] T 237c3af7733947e685cc90f998da736e P 7f4f6d95f2c14d5382a43b0fdedfee01: No bootstrap required, opened a new log
I20260924 03:26:32.061959 29056 tablet_bootstrap.cc:654] T 237c3af7733947e685cc90f998da736e P 5934bf6602ee4aa681d17199b28d2f39: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:32.062052 29054 ts_tablet_manager.cc:1397] T 237c3af7733947e685cc90f998da736e P 7f4f6d95f2c14d5382a43b0fdedfee01: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 03:26:32.063472 29056 log.cc:824] T 237c3af7733947e685cc90f998da736e P 5934bf6602ee4aa681d17199b28d2f39: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:32.064165 29055 tablet_bootstrap.cc:654] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:32.065629 29055 log.cc:824] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:32.069233 29055 tablet_bootstrap.cc:492] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455: No bootstrap required, opened a new log
I20260924 03:26:32.069762 29055 ts_tablet_manager.cc:1397] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.005s
I20260924 03:26:32.070092 29056 tablet_bootstrap.cc:492] T 237c3af7733947e685cc90f998da736e P 5934bf6602ee4aa681d17199b28d2f39: No bootstrap required, opened a new log
I20260924 03:26:32.070384 29056 ts_tablet_manager.cc:1397] T 237c3af7733947e685cc90f998da736e P 5934bf6602ee4aa681d17199b28d2f39: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.009s
I20260924 03:26:32.070865 29054 raft_consensus.cc:348] T 237c3af7733947e685cc90f998da736e P 7f4f6d95f2c14d5382a43b0fdedfee01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b3aefe667449adab5e6821e840a455" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 45853 } } peers { permanent_uuid: "7f4f6d95f2c14d5382a43b0fdedfee01" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34273 } } peers { permanent_uuid: "5934bf6602ee4aa681d17199b28d2f39" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45179 } }
I20260924 03:26:32.071260 29054 raft_consensus.cc:374] T 237c3af7733947e685cc90f998da736e P 7f4f6d95f2c14d5382a43b0fdedfee01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:32.071353 29054 raft_consensus.cc:729] T 237c3af7733947e685cc90f998da736e P 7f4f6d95f2c14d5382a43b0fdedfee01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f4f6d95f2c14d5382a43b0fdedfee01, State: Initialized, Role: FOLLOWER
I20260924 03:26:32.072134 29054 consensus_queue.cc:260] T 237c3af7733947e685cc90f998da736e P 7f4f6d95f2c14d5382a43b0fdedfee01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b3aefe667449adab5e6821e840a455" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 45853 } } peers { permanent_uuid: "7f4f6d95f2c14d5382a43b0fdedfee01" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34273 } } peers { permanent_uuid: "5934bf6602ee4aa681d17199b28d2f39" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45179 } }
I20260924 03:26:32.073984 29054 ts_tablet_manager.cc:1428] T 237c3af7733947e685cc90f998da736e P 7f4f6d95f2c14d5382a43b0fdedfee01: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20260924 03:26:32.081178 29055 raft_consensus.cc:348] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b3aefe667449adab5e6821e840a455" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 45853 } } peers { permanent_uuid: "7f4f6d95f2c14d5382a43b0fdedfee01" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34273 } } peers { permanent_uuid: "5934bf6602ee4aa681d17199b28d2f39" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45179 } }
I20260924 03:26:32.081738 29055 raft_consensus.cc:374] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:32.081848 29055 raft_consensus.cc:729] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b3aefe667449adab5e6821e840a455, State: Initialized, Role: FOLLOWER
I20260924 03:26:32.081949 29056 raft_consensus.cc:348] T 237c3af7733947e685cc90f998da736e P 5934bf6602ee4aa681d17199b28d2f39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b3aefe667449adab5e6821e840a455" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 45853 } } peers { permanent_uuid: "7f4f6d95f2c14d5382a43b0fdedfee01" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34273 } } peers { permanent_uuid: "5934bf6602ee4aa681d17199b28d2f39" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45179 } }
I20260924 03:26:32.082362 29056 raft_consensus.cc:374] T 237c3af7733947e685cc90f998da736e P 5934bf6602ee4aa681d17199b28d2f39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:32.082463 29056 raft_consensus.cc:729] T 237c3af7733947e685cc90f998da736e P 5934bf6602ee4aa681d17199b28d2f39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5934bf6602ee4aa681d17199b28d2f39, State: Initialized, Role: FOLLOWER
I20260924 03:26:32.082566 29055 consensus_queue.cc:260] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b3aefe667449adab5e6821e840a455" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 45853 } } peers { permanent_uuid: "7f4f6d95f2c14d5382a43b0fdedfee01" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34273 } } peers { permanent_uuid: "5934bf6602ee4aa681d17199b28d2f39" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45179 } }
I20260924 03:26:32.083045 29056 consensus_queue.cc:260] T 237c3af7733947e685cc90f998da736e P 5934bf6602ee4aa681d17199b28d2f39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b3aefe667449adab5e6821e840a455" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 45853 } } peers { permanent_uuid: "7f4f6d95f2c14d5382a43b0fdedfee01" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34273 } } peers { permanent_uuid: "5934bf6602ee4aa681d17199b28d2f39" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45179 } }
I20260924 03:26:32.083773 29035 heartbeater.cc:503] Master 127.10.12.190:32897 was elected leader, sending a full tablet report...
I20260924 03:26:32.084254 28901 heartbeater.cc:503] Master 127.10.12.190:32897 was elected leader, sending a full tablet report...
I20260924 03:26:32.084604 29055 ts_tablet_manager.cc:1428] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260924 03:26:32.084758 29056 ts_tablet_manager.cc:1428] T 237c3af7733947e685cc90f998da736e P 5934bf6602ee4aa681d17199b28d2f39: Time spent starting tablet: real 0.014s user 0.010s sys 0.006s
W20260924 03:26:32.110548 28902 tablet.cc:2369] T 237c3af7733947e685cc90f998da736e P 5934bf6602ee4aa681d17199b28d2f39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:26:32.147043 29061 raft_consensus.cc:482] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:26:32.147459 29061 raft_consensus.cc:504] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b3aefe667449adab5e6821e840a455" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 45853 } } peers { permanent_uuid: "7f4f6d95f2c14d5382a43b0fdedfee01" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34273 } } peers { permanent_uuid: "5934bf6602ee4aa681d17199b28d2f39" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45179 } }
I20260924 03:26:32.149308 29061 leader_election.cc:288] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f4f6d95f2c14d5382a43b0fdedfee01 (127.10.12.129:34273), 5934bf6602ee4aa681d17199b28d2f39 (127.10.12.130:45179)
I20260924 03:26:32.156260 29052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:32.149808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:39459 (local address 127.10.12.129:34273)
0924 03:26:32.150035 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:32.150041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:32.150532 (+ 491us) server_negotiation.cc:407] Connection header received
0924 03:26:32.150623 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:32.150629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:32.150707 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:32.150803 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:32.151251 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.152194 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.154114 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.155289 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.155428 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:32.155486 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:32.155892 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 03:26:32.156050 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 03:26:32.156339 29038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:32.149820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:34273 (local address 127.10.12.131:39459)
0924 03:26:32.150189 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:32.150209 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:32.150397 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:32.150852 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:32.150859 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:32.150893 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:32.151084 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:32.151096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.152339 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.152345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:32.153921 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:32.153936 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.155478 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.155484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:32.155711 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:32.155749 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 03:26:32.155849 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 03:26:32.156029 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":1}
I20260924 03:26:32.157238 29051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:32.149992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:36657 (local address 127.10.12.130:45179)
0924 03:26:32.150203 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:32.150209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:32.150916 (+ 707us) server_negotiation.cc:407] Connection header received
0924 03:26:32.151180 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:32.151186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:32.151263 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:32.151345 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:32.151928 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.152887 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.155025 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.156176 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.156323 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:32.156406 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:32.156848 (+ 442us) server_negotiation.cc:299] Negotiation successful
0924 03:26:32.156994 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 03:26:32.157461 29063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:32.149866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:45179 (local address 127.10.12.131:36657)
0924 03:26:32.150825 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:32.150843 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:32.151012 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:32.151372 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:32.151377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:32.151400 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:32.151757 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:32.151770 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.153067 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.153071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:32.154832 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:32.154849 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.156488 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.156494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:32.156667 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:32.156697 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 03:26:32.156875 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 03:26:32.157007 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":22,"thread_start_us":575,"threads_started":1}
I20260924 03:26:32.157761 28722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237c3af7733947e685cc90f998da736e" candidate_uuid: "13b3aefe667449adab5e6821e840a455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f4f6d95f2c14d5382a43b0fdedfee01" is_pre_election: true
I20260924 03:26:32.158327 28722 raft_consensus.cc:2454] T 237c3af7733947e685cc90f998da736e P 7f4f6d95f2c14d5382a43b0fdedfee01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13b3aefe667449adab5e6821e840a455 in term 0.
I20260924 03:26:32.158476 28856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237c3af7733947e685cc90f998da736e" candidate_uuid: "13b3aefe667449adab5e6821e840a455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5934bf6602ee4aa681d17199b28d2f39" is_pre_election: true
I20260924 03:26:32.158957 28856 raft_consensus.cc:2454] T 237c3af7733947e685cc90f998da736e P 5934bf6602ee4aa681d17199b28d2f39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13b3aefe667449adab5e6821e840a455 in term 0.
I20260924 03:26:32.159109 28923 leader_election.cc:302] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13b3aefe667449adab5e6821e840a455, 7f4f6d95f2c14d5382a43b0fdedfee01; no voters:
I20260924 03:26:32.159516 29061 raft_consensus.cc:2781] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:26:32.159654 29061 raft_consensus.cc:482] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:26:32.159754 29061 raft_consensus.cc:3037] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:32.162061 29061 raft_consensus.cc:504] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b3aefe667449adab5e6821e840a455" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 45853 } } peers { permanent_uuid: "7f4f6d95f2c14d5382a43b0fdedfee01" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34273 } } peers { permanent_uuid: "5934bf6602ee4aa681d17199b28d2f39" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45179 } }
I20260924 03:26:32.162726 29061 leader_election.cc:288] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455 [CANDIDATE]: Term 1 election: Requested vote from peers 7f4f6d95f2c14d5382a43b0fdedfee01 (127.10.12.129:34273), 5934bf6602ee4aa681d17199b28d2f39 (127.10.12.130:45179)
I20260924 03:26:32.163338 28856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237c3af7733947e685cc90f998da736e" candidate_uuid: "13b3aefe667449adab5e6821e840a455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5934bf6602ee4aa681d17199b28d2f39"
I20260924 03:26:32.163396 28722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "237c3af7733947e685cc90f998da736e" candidate_uuid: "13b3aefe667449adab5e6821e840a455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f4f6d95f2c14d5382a43b0fdedfee01"
I20260924 03:26:32.163559 28856 raft_consensus.cc:3037] T 237c3af7733947e685cc90f998da736e P 5934bf6602ee4aa681d17199b28d2f39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:32.163609 28722 raft_consensus.cc:3037] T 237c3af7733947e685cc90f998da736e P 7f4f6d95f2c14d5382a43b0fdedfee01 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:32.165939 28722 raft_consensus.cc:2454] T 237c3af7733947e685cc90f998da736e P 7f4f6d95f2c14d5382a43b0fdedfee01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13b3aefe667449adab5e6821e840a455 in term 1.
I20260924 03:26:32.165938 28856 raft_consensus.cc:2454] T 237c3af7733947e685cc90f998da736e P 5934bf6602ee4aa681d17199b28d2f39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13b3aefe667449adab5e6821e840a455 in term 1.
I20260924 03:26:32.166414 28923 leader_election.cc:302] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13b3aefe667449adab5e6821e840a455, 7f4f6d95f2c14d5382a43b0fdedfee01; no voters:
I20260924 03:26:32.166750 29061 raft_consensus.cc:2781] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:26:32.167218 29061 raft_consensus.cc:686] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455 [term 1 LEADER]: Becoming Leader. State: Replica: 13b3aefe667449adab5e6821e840a455, State: Running, Role: LEADER
I20260924 03:26:32.167784 29061 consensus_queue.cc:237] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b3aefe667449adab5e6821e840a455" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 45853 } } peers { permanent_uuid: "7f4f6d95f2c14d5382a43b0fdedfee01" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34273 } } peers { permanent_uuid: "5934bf6602ee4aa681d17199b28d2f39" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45179 } }
W20260924 03:26:32.171650 29036 tablet.cc:2369] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:26:32.172640 28577 catalog_manager.cc:5495] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13b3aefe667449adab5e6821e840a455 (127.10.12.131). New cstate: current_term: 1 leader_uuid: "13b3aefe667449adab5e6821e840a455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b3aefe667449adab5e6821e840a455" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 45853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f4f6d95f2c14d5382a43b0fdedfee01" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5934bf6602ee4aa681d17199b28d2f39" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20260924 03:26:32.195621 29049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:32.189053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44676 (local address 127.10.12.131:45853)
0924 03:26:32.189268 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:32.189273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:32.189290 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:26:32.189487 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:32.189491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:32.189571 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:32.189698 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:26:32.190066 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.190847 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.192603 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.193031 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.193142 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:32.194104 (+ 962us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:26:32.194707 (+ 603us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:26:32.194854 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:32.195311 (+ 457us) server_negotiation.cc:299] Negotiation successful
0924 03:26:32.195365 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 03:26:32.204490 28722 raft_consensus.cc:1261] T 237c3af7733947e685cc90f998da736e P 7f4f6d95f2c14d5382a43b0fdedfee01 [term 1 FOLLOWER]: Refusing update from remote peer 13b3aefe667449adab5e6821e840a455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:26:32.204490 28856 raft_consensus.cc:1261] T 237c3af7733947e685cc90f998da736e P 5934bf6602ee4aa681d17199b28d2f39 [term 1 FOLLOWER]: Refusing update from remote peer 13b3aefe667449adab5e6821e840a455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:26:32.205744 29061 consensus_queue.cc:1035] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f4f6d95f2c14d5382a43b0fdedfee01" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 34273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:26:32.206079 29064 consensus_queue.cc:1035] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5934bf6602ee4aa681d17199b28d2f39" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:26:32.221680 29068 mvcc.cc:205] Tried to move back new op lower bound from 7332742726458564608 to 7332742726323613696. Current Snapshot: MvccSnapshot[applied={T|T < 7332742726458564608}]
I20260924 03:26:32.226744 29069 mvcc.cc:205] Tried to move back new op lower bound from 7332742726458564608 to 7332742726323613696. Current Snapshot: MvccSnapshot[applied={T|T < 7332742726458564608}]
I20260924 03:26:32.237025 28577 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:26:32.237380 28577 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:26:32.242393 28577 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3566ae82ebde4f39b82b84eeca056dbe: Sending DeleteTablet for 3 replicas of tablet 237c3af7733947e685cc90f998da736e
I20260924 03:26:32.243907 28702 tablet_service.cc:1514] Processing DeleteTablet for tablet 237c3af7733947e685cc90f998da736e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:26:32 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:44092
I20260924 03:26:32.243908 28836 tablet_service.cc:1514] Processing DeleteTablet for tablet 237c3af7733947e685cc90f998da736e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:26:32 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:45828
I20260924 03:26:32.245106 29077 tablet_replica.cc:330] T 237c3af7733947e685cc90f998da736e P 7f4f6d95f2c14d5382a43b0fdedfee01: stopping tablet replica
I20260924 03:26:32.245225 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 28639
I20260924 03:26:32.245141 28970 tablet_service.cc:1514] Processing DeleteTablet for tablet 237c3af7733947e685cc90f998da736e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:26:32 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:44670
I20260924 03:26:32.245572 29077 raft_consensus.cc:2229] T 237c3af7733947e685cc90f998da736e P 7f4f6d95f2c14d5382a43b0fdedfee01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:26:32.246129 29077 raft_consensus.cc:2258] T 237c3af7733947e685cc90f998da736e P 7f4f6d95f2c14d5382a43b0fdedfee01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:26:32.246928 29078 tablet_replica.cc:330] T 237c3af7733947e685cc90f998da736e P 5934bf6602ee4aa681d17199b28d2f39: stopping tablet replica
I20260924 03:26:32.247354 29078 raft_consensus.cc:2229] T 237c3af7733947e685cc90f998da736e P 5934bf6602ee4aa681d17199b28d2f39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:26:32.247875 29078 raft_consensus.cc:2258] T 237c3af7733947e685cc90f998da736e P 5934bf6602ee4aa681d17199b28d2f39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:26:32.248751 29079 tablet_replica.cc:330] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455: stopping tablet replica
I20260924 03:26:32.249162 29079 raft_consensus.cc:2229] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:26:32.249958 29079 raft_consensus.cc:2258] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:26:32.250953 29077 ts_tablet_manager.cc:1905] T 237c3af7733947e685cc90f998da736e P 7f4f6d95f2c14d5382a43b0fdedfee01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:26:32.251071 29078 ts_tablet_manager.cc:1905] T 237c3af7733947e685cc90f998da736e P 5934bf6602ee4aa681d17199b28d2f39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:26:32.258674 29078 ts_tablet_manager.cc:1918] T 237c3af7733947e685cc90f998da736e P 5934bf6602ee4aa681d17199b28d2f39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:26:32.258898 29078 log.cc:1196] T 237c3af7733947e685cc90f998da736e P 5934bf6602ee4aa681d17199b28d2f39: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/237c3af7733947e685cc90f998da736e
I20260924 03:26:32.259436 29078 ts_tablet_manager.cc:1939] T 237c3af7733947e685cc90f998da736e P 5934bf6602ee4aa681d17199b28d2f39: Deleting consensus metadata
I20260924 03:26:32.259459 29079 ts_tablet_manager.cc:1905] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:26:32.261536 28565 catalog_manager.cc:4841] TS 5934bf6602ee4aa681d17199b28d2f39 (127.10.12.130:45179): tablet 237c3af7733947e685cc90f998da736e (table test-table [id=f1474885976b49af8e4628b5d139f709]) successfully deleted
I20260924 03:26:32.264374 29077 ts_tablet_manager.cc:1918] T 237c3af7733947e685cc90f998da736e P 7f4f6d95f2c14d5382a43b0fdedfee01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:26:32.264596 29077 log.cc:1196] T 237c3af7733947e685cc90f998da736e P 7f4f6d95f2c14d5382a43b0fdedfee01: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-0/wal/wals/237c3af7733947e685cc90f998da736e
I20260924 03:26:32.265365 29077 ts_tablet_manager.cc:1939] T 237c3af7733947e685cc90f998da736e P 7f4f6d95f2c14d5382a43b0fdedfee01: Deleting consensus metadata
I20260924 03:26:32.267758 28562 catalog_manager.cc:4841] TS 7f4f6d95f2c14d5382a43b0fdedfee01 (127.10.12.129:34273): tablet 237c3af7733947e685cc90f998da736e (table test-table [id=f1474885976b49af8e4628b5d139f709]) successfully deleted
I20260924 03:26:32.268725 29079 ts_tablet_manager.cc:1918] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:26:32.269234 29079 log.cc:1196] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/237c3af7733947e685cc90f998da736e
I20260924 03:26:32.269879 29079 ts_tablet_manager.cc:1939] T 237c3af7733947e685cc90f998da736e P 13b3aefe667449adab5e6821e840a455: Deleting consensus metadata
I20260924 03:26:32.271917 28562 catalog_manager.cc:4841] TS 13b3aefe667449adab5e6821e840a455 (127.10.12.131:45853): tablet 237c3af7733947e685cc90f998da736e (table test-table [id=f1474885976b49af8e4628b5d139f709]) successfully deleted
Sep 24 03:26:32 dist-test-slave-00qs krb5kdc[28533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.129@KRBTEST.COM
I20260924 03:26:32.281443 29052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:32.248775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44104 (local address 127.10.12.129:34273)
0924 03:26:32.249079 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:32.249086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:32.262281 (+ 13195us) server_negotiation.cc:407] Connection header received
0924 03:26:32.262496 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:32.262502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:32.262599 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:32.262672 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:32.264664 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.265805 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.268251 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.268886 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.268990 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:32.274823 (+ 5833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:32.274843 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:32.274860 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:32.274906 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:32.278301 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:32.279158 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:32.279167 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:32.279169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:32.279324 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:32.279716 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:32.279720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:32.279722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:32.280214 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:32.280526 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:32.280839 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 03:26:32.281085 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260924 03:26:32.282241 28762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:32.412391 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 28639
I20260924 03:26:32.428251 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 28773
Sep 24 03:26:32 dist-test-slave-00qs krb5kdc[28533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.130@KRBTEST.COM
I20260924 03:26:32.448547 29051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:32.429090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45836 (local address 127.10.12.130:45179)
0924 03:26:32.429319 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:32.429326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:32.429349 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:26:32.429418 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:32.429424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:32.429516 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:32.429630 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:32.431024 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.432278 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.433712 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.434547 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.434772 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:32.439999 (+ 5227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:32.440023 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:32.440044 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:32.440098 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:32.444454 (+ 4356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:32.445953 (+ 1499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:32.445965 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:32.445976 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:32.446169 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:32.446666 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:32.446671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:32.446674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:32.447209 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:32.447530 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:32.447871 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 03:26:32.448190 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 03:26:32.449520 28896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:32.541131 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 28773
I20260924 03:26:32.562357 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 28907
Sep 24 03:26:32 dist-test-slave-00qs krb5kdc[28533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.131@KRBTEST.COM
I20260924 03:26:32.582289 29049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:32.563139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44684 (local address 127.10.12.131:45853)
0924 03:26:32.563349 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:32.563355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:32.563377 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:26:32.563450 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:32.563456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:32.563533 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:32.563646 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:32.564992 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.566192 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.567759 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:32.568477 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:32.568654 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:32.573830 (+ 5176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:32.573855 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:32.573879 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:32.573946 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:32.578113 (+ 4167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:32.579560 (+ 1447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:32.579572 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:32.579576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:32.579820 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:32.580471 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:32.580477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:32.580480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:32.581025 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:32.581386 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:32.581638 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 03:26:32.581910 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 03:26:32.583094 29030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:32.710520 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 28907
I20260924 03:26:32.727125 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 28546
I20260924 03:26:32.728286 28608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:32.842787 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 28546
2026-09-24T03:26:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4621 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T03:26:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:26:32Z Disabled control of system clock
I20260924 03:26:32.916800 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:45999
--webserver_interface=127.10.12.190
--webserver_port=0
--builtin_ntp_servers=127.10.12.148:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:45999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:26:33.273054 29088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:33.273393 29088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:33.273485 29088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:33.281311 29088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:26:33.281457 29088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:33.281497 29088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:33.281576 29088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:26:33.281638 29088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:26:33.291472 29088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:45999
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:45999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:33.292804 29088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:33.294634 29088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:33.307276 29096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:33.307310 29094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:33.307327 29093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:33.308102 29088 server_base.cc:1034] running on GCE node
I20260924 03:26:33.309010 29088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:33.310696 29088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:33.311939 29088 hybrid_clock.cc:648] HybridClock initialized: now 1790220393311866 us; error 63 us; skew 500 ppm
I20260924 03:26:33.312374 29088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:26:33.315109 29088 webserver.cc:460] Webserver started at http://127.10.12.190:42991/ using document root <none> and password file <none>
I20260924 03:26:33.315874 29088 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:33.316012 29088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:33.316335 29088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:33.319386 29088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "f95701f3d13444499426a91341a01b79"
format_stamp: "Formatted at 2026-09-24 03:26:33 on dist-test-slave-00qs"
server_key: "9575a6ddba4fd32da0d970bf134edfc5"
server_key_iv: "94be536ae15e7b8ddb957026da2b934d"
server_key_version: "encryptionkey@0"
I20260924 03:26:33.320289 29088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "f95701f3d13444499426a91341a01b79"
format_stamp: "Formatted at 2026-09-24 03:26:33 on dist-test-slave-00qs"
server_key: "9575a6ddba4fd32da0d970bf134edfc5"
server_key_iv: "94be536ae15e7b8ddb957026da2b934d"
server_key_version: "encryptionkey@0"
I20260924 03:26:33.325893 29088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:26:33.329905 29102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:33.331688 29088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:26:33.331902 29088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "f95701f3d13444499426a91341a01b79"
format_stamp: "Formatted at 2026-09-24 03:26:33 on dist-test-slave-00qs"
server_key: "9575a6ddba4fd32da0d970bf134edfc5"
server_key_iv: "94be536ae15e7b8ddb957026da2b934d"
server_key_version: "encryptionkey@0"
I20260924 03:26:33.332165 29088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:33.386976 29088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:33.387982 29088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:33.388291 29088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:33.407603 29088 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:45999
I20260924 03:26:33.407608 29153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:45999 every 8 connection(s)
I20260924 03:26:33.409025 29088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:26:33.413295 29154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:33.413350 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 29088
I20260924 03:26:33.413511 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:26:33.414088 10290 external_mini_cluster.cc:1428] Setting key bf5f8cf79065f9078af35a953964f5ef
I20260924 03:26:33.427570 29154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f95701f3d13444499426a91341a01b79: Bootstrap starting.
I20260924 03:26:33.429507 29157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:33.416254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36684 (local address 127.10.12.190:45999)
0924 03:26:33.417355 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:33.417373 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:33.417421 (+ 48us) server_negotiation.cc:407] Connection header received
0924 03:26:33.419702 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:33.419730 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:33.420358 (+ 628us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:33.420721 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:33.421389 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:33.422708 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:33.424709 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:33.425518 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:33.425663 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:33.427004 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:33.427012 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:33.427028 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:33.427602 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:33.427721 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:33.427753 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:33.427853 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:33.428173 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 03:26:33.428520 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":403,"threads_started":1}
I20260924 03:26:33.431146 29154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f95701f3d13444499426a91341a01b79: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:33.432557 29154 log.cc:824] T 00000000000000000000000000000000 P f95701f3d13444499426a91341a01b79: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:33.435513 29154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f95701f3d13444499426a91341a01b79: No bootstrap required, opened a new log
I20260924 03:26:33.443181 29154 raft_consensus.cc:348] T 00000000000000000000000000000000 P f95701f3d13444499426a91341a01b79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95701f3d13444499426a91341a01b79" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45999 } }
I20260924 03:26:33.443591 29154 raft_consensus.cc:374] T 00000000000000000000000000000000 P f95701f3d13444499426a91341a01b79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:33.443684 29154 raft_consensus.cc:729] T 00000000000000000000000000000000 P f95701f3d13444499426a91341a01b79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f95701f3d13444499426a91341a01b79, State: Initialized, Role: FOLLOWER
I20260924 03:26:33.444334 29154 consensus_queue.cc:260] T 00000000000000000000000000000000 P f95701f3d13444499426a91341a01b79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95701f3d13444499426a91341a01b79" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45999 } }
I20260924 03:26:33.444574 29154 raft_consensus.cc:388] T 00000000000000000000000000000000 P f95701f3d13444499426a91341a01b79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:26:33.444710 29154 raft_consensus.cc:482] T 00000000000000000000000000000000 P f95701f3d13444499426a91341a01b79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:26:33.444877 29154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f95701f3d13444499426a91341a01b79 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:33.447450 29154 raft_consensus.cc:504] T 00000000000000000000000000000000 P f95701f3d13444499426a91341a01b79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95701f3d13444499426a91341a01b79" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45999 } }
I20260924 03:26:33.448009 29154 leader_election.cc:302] T 00000000000000000000000000000000 P f95701f3d13444499426a91341a01b79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f95701f3d13444499426a91341a01b79; no voters:
I20260924 03:26:33.448557 29154 leader_election.cc:288] T 00000000000000000000000000000000 P f95701f3d13444499426a91341a01b79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:26:33.448710 29159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f95701f3d13444499426a91341a01b79 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:26:33.449493 29159 raft_consensus.cc:686] T 00000000000000000000000000000000 P f95701f3d13444499426a91341a01b79 [term 1 LEADER]: Becoming Leader. State: Replica: f95701f3d13444499426a91341a01b79, State: Running, Role: LEADER
I20260924 03:26:33.449996 29159 consensus_queue.cc:237] T 00000000000000000000000000000000 P f95701f3d13444499426a91341a01b79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95701f3d13444499426a91341a01b79" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45999 } }
I20260924 03:26:33.450701 29154 sys_catalog.cc:564] T 00000000000000000000000000000000 P f95701f3d13444499426a91341a01b79 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:26:33.452500 29160 sys_catalog.cc:455] T 00000000000000000000000000000000 P f95701f3d13444499426a91341a01b79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f95701f3d13444499426a91341a01b79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95701f3d13444499426a91341a01b79" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45999 } } }
I20260924 03:26:33.452931 29160 sys_catalog.cc:458] T 00000000000000000000000000000000 P f95701f3d13444499426a91341a01b79 [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:33.454007 29161 sys_catalog.cc:455] T 00000000000000000000000000000000 P f95701f3d13444499426a91341a01b79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f95701f3d13444499426a91341a01b79. Latest consensus state: current_term: 1 leader_uuid: "f95701f3d13444499426a91341a01b79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95701f3d13444499426a91341a01b79" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45999 } } }
I20260924 03:26:33.454413 29161 sys_catalog.cc:458] T 00000000000000000000000000000000 P f95701f3d13444499426a91341a01b79 [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:33.454738 29165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:26:33.460758 29165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:26:33.470978 29165 catalog_manager.cc:1347] Generated new cluster ID: 6399a4e75be342a98f3f230558aa27dc
I20260924 03:26:33.471108 29165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:26:33.481199 29165 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:26:33.482679 29165 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:26:33.490108 29165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f95701f3d13444499426a91341a01b79: Generated new TSK 0
I20260924 03:26:33.491109 29165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:26:33.505513 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:45999
--builtin_ntp_servers=127.10.12.148:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:26:33.849376 29178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:33.849668 29178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:33.849776 29178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:33.857425 29178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:33.857542 29178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:33.857676 29178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:26:33.867113 29178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:45999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:33.868862 29178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:33.870849 29178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:33.884590 29183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:33.884625 29186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:33.884647 29184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:33.885270 29178 server_base.cc:1034] running on GCE node
I20260924 03:26:33.886013 29178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:33.887073 29178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:33.888302 29178 hybrid_clock.cc:648] HybridClock initialized: now 1790220393888233 us; error 63 us; skew 500 ppm
I20260924 03:26:33.888691 29178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:26:33.891211 29178 webserver.cc:460] Webserver started at http://127.10.12.129:38785/ using document root <none> and password file <none>
I20260924 03:26:33.891943 29178 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:33.892088 29178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:33.892401 29178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:33.895411 29178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "952dda47b0514e98a5fe41ff82b53748"
format_stamp: "Formatted at 2026-09-24 03:26:33 on dist-test-slave-00qs"
server_key: "1856499d0247b6932af335d7b653c4e7"
server_key_iv: "4a686b9fafc1a49294e41d2d30d2faa0"
server_key_version: "encryptionkey@0"
I20260924 03:26:33.896293 29178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "952dda47b0514e98a5fe41ff82b53748"
format_stamp: "Formatted at 2026-09-24 03:26:33 on dist-test-slave-00qs"
server_key: "1856499d0247b6932af335d7b653c4e7"
server_key_iv: "4a686b9fafc1a49294e41d2d30d2faa0"
server_key_version: "encryptionkey@0"
I20260924 03:26:33.902073 29178 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:26:33.906087 29192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:33.907569 29178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:26:33.907745 29178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "952dda47b0514e98a5fe41ff82b53748"
format_stamp: "Formatted at 2026-09-24 03:26:33 on dist-test-slave-00qs"
server_key: "1856499d0247b6932af335d7b653c4e7"
server_key_iv: "4a686b9fafc1a49294e41d2d30d2faa0"
server_key_version: "encryptionkey@0"
I20260924 03:26:33.907991 29178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:33.926210 29178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:33.927086 29178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:33.927373 29178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:33.928674 29178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:26:33.930488 29178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:26:33.930585 29178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:33.930665 29178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:26:33.930716 29178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:33.961860 29178 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:46155
I20260924 03:26:33.961951 29304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:46155 every 8 connection(s)
I20260924 03:26:33.963253 29178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:26:33.971482 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 29178
I20260924 03:26:33.971668 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:26:33.972216 10290 external_mini_cluster.cc:1428] Setting key 327c63b7286d9cb900d91ffd9c79eecd
I20260924 03:26:33.975776 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:45999
--builtin_ntp_servers=127.10.12.148:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:26:33.979041 29307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:33.965977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:52261 (local address 127.10.12.190:45999)
0924 03:26:33.966509 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:33.966514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:33.967516 (+ 1002us) server_negotiation.cc:407] Connection header received
0924 03:26:33.969378 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:33.969384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:33.969481 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:33.969579 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:33.970760 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:33.971726 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:33.975064 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:33.975697 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:33.975835 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:33.977723 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:33.977730 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:33.977734 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:33.977759 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:33.977819 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:33.977836 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:33.977919 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:33.978258 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 03:26:33.978431 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":256,"threads_started":1}
I20260924 03:26:33.980229 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:33.966419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:45999 (local address 127.10.12.129:52261)
0924 03:26:33.967322 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:33.967363 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:33.969087 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:33.969837 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:33.969846 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:33.969950 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:26:33.970578 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:33.970596 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:33.972285 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:33.972295 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:33.974834 (+ 2539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:33.974861 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:33.976036 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:33.976074 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:33.976496 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:33.977276 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:26:33.977280 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:26:33.977349 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:26:33.977465 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:26:33.977971 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:26:33.977983 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:26:33.977992 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 03:26:33.978183 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 03:26:33.978647 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":265,"threads_started":1}
I20260924 03:26:33.983206 29305 heartbeater.cc:347] Connected to a master server at 127.10.12.190:45999
I20260924 03:26:33.983564 29305 heartbeater.cc:464] Registering TS with master...
I20260924 03:26:33.984617 29305 heartbeater.cc:511] Master 127.10.12.190:45999 requested a full tablet report, sending...
I20260924 03:26:33.986967 29119 ts_manager.cc:194] Registered new tserver with Master: 952dda47b0514e98a5fe41ff82b53748 (127.10.12.129:46155)
I20260924 03:26:33.988597 29119 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.12.129:52261
W20260924 03:26:34.310871 29309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:34.311192 29309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:34.311312 29309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:34.319134 29309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:34.319280 29309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:34.319440 29309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:26:34.329062 29309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:45999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:34.330463 29309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:34.332386 29309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:34.345685 29314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:34.345825 29317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:34.345876 29315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:34.346504 29309 server_base.cc:1034] running on GCE node
I20260924 03:26:34.347159 29309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:34.348157 29309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:34.349392 29309 hybrid_clock.cc:648] HybridClock initialized: now 1790220394349341 us; error 41 us; skew 500 ppm
I20260924 03:26:34.349726 29309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:26:34.352124 29309 webserver.cc:460] Webserver started at http://127.10.12.130:44313/ using document root <none> and password file <none>
I20260924 03:26:34.352806 29309 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:34.352906 29309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:34.353211 29309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:34.356298 29309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "59284551533f47e9b56e5a1087371e42"
format_stamp: "Formatted at 2026-09-24 03:26:34 on dist-test-slave-00qs"
server_key: "f0515d3ae595e9ad9a2b49b519df69d2"
server_key_iv: "75d42c6bba0270676a1748dd64899e7b"
server_key_version: "encryptionkey@0"
I20260924 03:26:34.357157 29309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "59284551533f47e9b56e5a1087371e42"
format_stamp: "Formatted at 2026-09-24 03:26:34 on dist-test-slave-00qs"
server_key: "f0515d3ae595e9ad9a2b49b519df69d2"
server_key_iv: "75d42c6bba0270676a1748dd64899e7b"
server_key_version: "encryptionkey@0"
I20260924 03:26:34.362566 29309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 03:26:34.366309 29323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:34.367918 29309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:26:34.368178 29309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "59284551533f47e9b56e5a1087371e42"
format_stamp: "Formatted at 2026-09-24 03:26:34 on dist-test-slave-00qs"
server_key: "f0515d3ae595e9ad9a2b49b519df69d2"
server_key_iv: "75d42c6bba0270676a1748dd64899e7b"
server_key_version: "encryptionkey@0"
I20260924 03:26:34.368409 29309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:34.381336 29309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:34.382370 29309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:34.382772 29309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:34.383809 29309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:26:34.385454 29309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:26:34.385558 29309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:34.385674 29309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:26:34.385743 29309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:34.418054 29309 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:42647
I20260924 03:26:34.418088 29435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:42647 every 8 connection(s)
I20260924 03:26:34.419376 29309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:26:34.424232 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 29309
I20260924 03:26:34.424417 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:26:34.424995 10290 external_mini_cluster.cc:1428] Setting key da7b7710cfbfc387b001639f33f543f8
I20260924 03:26:34.428397 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:45999
--builtin_ntp_servers=127.10.12.148:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:26:34.434677 29307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:34.422117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:39891 (local address 127.10.12.190:45999)
0924 03:26:34.422287 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:34.422292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:34.423992 (+ 1700us) server_negotiation.cc:407] Connection header received
0924 03:26:34.426510 (+ 2518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:34.426515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:34.426591 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:34.426697 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:34.428195 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:34.429056 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:34.431550 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:34.432042 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:34.432161 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:34.433842 (+ 1681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:34.433849 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:34.433853 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:34.433876 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:34.433912 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:34.433927 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:34.434013 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:34.434281 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 03:26:34.434434 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 03:26:34.436141 29438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:34.422716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:45999 (local address 127.10.12.130:39891)
0924 03:26:34.423764 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:34.423827 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:34.426209 (+ 2382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:34.426916 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:34.426928 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:34.427073 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:26:34.427940 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:34.428005 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:34.429246 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:34.429252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:34.431270 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:34.431291 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:34.432279 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:34.432285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:34.432594 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:34.433386 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:26:34.433390 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:26:34.433464 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:26:34.433581 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:26:34.434036 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:26:34.434047 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:26:34.434055 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 03:26:34.434255 (+ 200us) client_negotiation.cc:240] Negotiation successful
0924 03:26:34.434702 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":343,"threads_started":1}
I20260924 03:26:34.437853 29436 heartbeater.cc:347] Connected to a master server at 127.10.12.190:45999
I20260924 03:26:34.438165 29436 heartbeater.cc:464] Registering TS with master...
I20260924 03:26:34.438925 29436 heartbeater.cc:511] Master 127.10.12.190:45999 requested a full tablet report, sending...
I20260924 03:26:34.440644 29119 ts_manager.cc:194] Registered new tserver with Master: 59284551533f47e9b56e5a1087371e42 (127.10.12.130:42647)
I20260924 03:26:34.441697 29119 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.12.130:39891
W20260924 03:26:34.762967 29439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:34.763454 29439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:34.763607 29439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:34.771557 29439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:34.771714 29439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:34.771863 29439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:26:34.781571 29439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:45999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:34.782948 29439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:34.784798 29439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:34.799058 29445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:34.799058 29444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:34.799059 29447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:34.799875 29439 server_base.cc:1034] running on GCE node
I20260924 03:26:34.800531 29439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:34.801532 29439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:34.802762 29439 hybrid_clock.cc:648] HybridClock initialized: now 1790220394802716 us; error 36 us; skew 500 ppm
I20260924 03:26:34.803238 29439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:26:34.805724 29439 webserver.cc:460] Webserver started at http://127.10.12.131:43561/ using document root <none> and password file <none>
I20260924 03:26:34.806447 29439 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:34.806540 29439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:34.806854 29439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:34.809945 29439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "3152be47624b416daf03651711f3fdc3"
format_stamp: "Formatted at 2026-09-24 03:26:34 on dist-test-slave-00qs"
server_key: "1f407fcd55526bd5d91372aba081abe1"
server_key_iv: "6671cc09f6921ed934c34e551fd1a90c"
server_key_version: "encryptionkey@0"
I20260924 03:26:34.810842 29439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "3152be47624b416daf03651711f3fdc3"
format_stamp: "Formatted at 2026-09-24 03:26:34 on dist-test-slave-00qs"
server_key: "1f407fcd55526bd5d91372aba081abe1"
server_key_iv: "6671cc09f6921ed934c34e551fd1a90c"
server_key_version: "encryptionkey@0"
I20260924 03:26:34.816855 29439 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
I20260924 03:26:34.821097 29453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:34.822791 29439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:26:34.822988 29439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "3152be47624b416daf03651711f3fdc3"
format_stamp: "Formatted at 2026-09-24 03:26:34 on dist-test-slave-00qs"
server_key: "1f407fcd55526bd5d91372aba081abe1"
server_key_iv: "6671cc09f6921ed934c34e551fd1a90c"
server_key_version: "encryptionkey@0"
I20260924 03:26:34.823313 29439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:34.838824 29439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:34.839675 29439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:34.840000 29439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:34.841102 29439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:26:34.842645 29439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:26:34.842736 29439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:34.842849 29439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:26:34.842914 29439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:34.875448 29439 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:34287
I20260924 03:26:34.875496 29565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:34287 every 8 connection(s)
I20260924 03:26:34.876830 29439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:26:34.887001 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 29439
I20260924 03:26:34.887185 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:26:34.887650 10290 external_mini_cluster.cc:1428] Setting key 356a55e77f7841fff33958818aab81cb
I20260924 03:26:34.890172 29307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:34.879579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:33409 (local address 127.10.12.190:45999)
0924 03:26:34.879799 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:34.879803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:34.881282 (+ 1479us) server_negotiation.cc:407] Connection header received
0924 03:26:34.883110 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:34.883116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:34.883207 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:34.883298 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:34.884565 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:34.885313 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:34.886909 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:34.887408 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:34.887527 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:34.889131 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:34.889138 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:34.889142 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:34.889165 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:34.889209 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:34.889227 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:34.889333 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:34.889758 (+ 425us) server_negotiation.cc:299] Negotiation successful
0924 03:26:34.889914 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 03:26:34.891705 29568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:34.880116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:45999 (local address 127.10.12.131:33409)
0924 03:26:34.881098 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:34.881134 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:34.882790 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:34.883580 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:34.883590 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:34.883691 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:26:34.884352 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:34.884370 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:34.885478 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:34.885482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:34.886739 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:34.886752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:34.887642 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:34.887650 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:34.887923 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:34.888772 (+ 849us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:26:34.888774 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:26:34.888820 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:26:34.888919 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:26:34.889471 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:26:34.889483 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:26:34.889491 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 03:26:34.889687 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 03:26:34.890220 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":343,"threads_started":1}
I20260924 03:26:34.893716 29566 heartbeater.cc:347] Connected to a master server at 127.10.12.190:45999
I20260924 03:26:34.894047 29566 heartbeater.cc:464] Registering TS with master...
I20260924 03:26:34.894814 29566 heartbeater.cc:511] Master 127.10.12.190:45999 requested a full tablet report, sending...
I20260924 03:26:34.896463 29119 ts_manager.cc:194] Registered new tserver with Master: 3152be47624b416daf03651711f3fdc3 (127.10.12.131:34287)
I20260924 03:26:34.897289 29119 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.12.131:33409
I20260924 03:26:34.904980 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:26:34.915828 29307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:34.909321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36690 (local address 127.10.12.190:45999)
0924 03:26:34.909565 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:34.909571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:34.909867 (+ 296us) server_negotiation.cc:407] Connection header received
0924 03:26:34.910106 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:34.910110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:34.910180 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:34.910270 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:34.910782 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:34.911521 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:34.913015 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:34.913559 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:34.913664 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:34.915000 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:34.915005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:34.915008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:34.915033 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:34.915070 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:34.915084 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:34.915203 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:34.915469 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 03:26:34.915534 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 03:26:34.920096 29119 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:26:34.927934 29119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:26:34.954458 29581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:34.942637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53450 (local address 127.10.12.129:46155)
0924 03:26:34.944281 (+ 1644us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:34.944291 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:34.944839 (+ 548us) server_negotiation.cc:407] Connection header received
0924 03:26:34.945181 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:34.945191 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:34.945474 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:34.945660 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:34.946624 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:34.948597 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:34.952192 (+ 3595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:34.953224 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:34.953359 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:34.953630 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:34.953922 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 03:26:34.954102 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1468,"thread_start_us":1338,"threads_started":1}
I20260924 03:26:34.954500 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:34.942073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:46155 (local address 127.0.0.1:53450)
0924 03:26:34.944726 (+ 2653us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:34.944745 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:34.944967 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:34.945979 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:34.945984 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:34.946014 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:34.946435 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:34.946446 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:34.948818 (+ 2372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:34.948825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:34.951920 (+ 3095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:34.951943 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:34.953439 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:34.953446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:34.953652 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:34.953711 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 03:26:34.953852 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 03:26:34.954063 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2510,"mutex_wait_us":581,"thread_start_us":722,"threads_started":1}
I20260924 03:26:34.955356 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:34.941443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:34287 (local address 127.0.0.1:41276)
0924 03:26:34.942899 (+ 1456us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:34.942944 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:34.943130 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:34.944544 (+ 1414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:34.944551 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:34.944797 (+ 246us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:34.945317 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:34.945331 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:34.948812 (+ 3481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:34.948820 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:34.952013 (+ 3193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:34.952030 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:34.954387 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:34.954394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:34.954699 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:34.954733 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 03:26:34.954821 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 03:26:34.954977 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1175,"mutex_wait_us":136,"thread_start_us":2973,"threads_started":1}
I20260924 03:26:34.956410 29579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:34.941554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41276 (local address 127.10.12.131:34287)
0924 03:26:34.943785 (+ 2231us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:34.943796 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:34.943835 (+ 39us) server_negotiation.cc:407] Connection header received
0924 03:26:34.943944 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:34.943977 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:34.944257 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:34.946169 (+ 1912us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:34.946485 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:34.948613 (+ 2128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:34.952262 (+ 3649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:34.953829 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:34.953993 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:34.954251 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:34.955784 (+ 1533us) server_negotiation.cc:299] Negotiation successful
0924 03:26:34.956048 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1944,"thread_start_us":1757,"threads_started":1}
I20260924 03:26:34.958926 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:34.944881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:42647 (local address 127.0.0.1:58794)
0924 03:26:34.949368 (+ 4487us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:34.949392 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:34.949521 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:34.950752 (+ 1231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:34.950757 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:34.950784 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:34.951329 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:34.951343 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:34.953673 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:34.953679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:34.955812 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:34.955827 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:34.958117 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:34.958124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:34.958301 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:34.958342 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 03:26:34.958425 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 03:26:34.958569 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4351,"mutex_wait_us":3,"thread_start_us":4180,"threads_started":1}
I20260924 03:26:34.959086 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:34.946425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58794 (local address 127.10.12.130:42647)
0924 03:26:34.950048 (+ 3623us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:34.950058 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:34.950092 (+ 34us) server_negotiation.cc:407] Connection header received
0924 03:26:34.950196 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:34.950206 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:34.950514 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:34.951608 (+ 1094us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:34.951879 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:34.953500 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:34.956069 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:34.957899 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:34.958038 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:34.958271 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:34.958493 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 03:26:34.958662 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3426,"thread_start_us":3585,"threads_started":1}
I20260924 03:26:34.959789 29240 tablet_service.cc:1467] Processing CreateTablet for tablet a9e04112036047bfb17de822465e9b26 (DEFAULT_TABLE table=test-table [id=e416e06a2c9c45d693195abd4fdfcdd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:26:34.959861 29501 tablet_service.cc:1467] Processing CreateTablet for tablet a9e04112036047bfb17de822465e9b26 (DEFAULT_TABLE table=test-table [id=e416e06a2c9c45d693195abd4fdfcdd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:26:34.961689 29240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e04112036047bfb17de822465e9b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:34.962358 29371 tablet_service.cc:1467] Processing CreateTablet for tablet a9e04112036047bfb17de822465e9b26 (DEFAULT_TABLE table=test-table [id=e416e06a2c9c45d693195abd4fdfcdd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:26:34.963483 29371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e04112036047bfb17de822465e9b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:34.965619 29501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e04112036047bfb17de822465e9b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:34.980379 29584 tablet_bootstrap.cc:492] T a9e04112036047bfb17de822465e9b26 P 952dda47b0514e98a5fe41ff82b53748: Bootstrap starting.
I20260924 03:26:34.982728 29585 tablet_bootstrap.cc:492] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42: Bootstrap starting.
I20260924 03:26:34.984380 29584 tablet_bootstrap.cc:654] T a9e04112036047bfb17de822465e9b26 P 952dda47b0514e98a5fe41ff82b53748: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:34.985720 29584 log.cc:824] T a9e04112036047bfb17de822465e9b26 P 952dda47b0514e98a5fe41ff82b53748: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:34.986598 29586 tablet_bootstrap.cc:492] T a9e04112036047bfb17de822465e9b26 P 3152be47624b416daf03651711f3fdc3: Bootstrap starting.
I20260924 03:26:34.986619 29585 tablet_bootstrap.cc:654] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:34.988204 29585 log.cc:824] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:34.990249 29584 tablet_bootstrap.cc:492] T a9e04112036047bfb17de822465e9b26 P 952dda47b0514e98a5fe41ff82b53748: No bootstrap required, opened a new log
I20260924 03:26:34.990605 29584 ts_tablet_manager.cc:1397] T a9e04112036047bfb17de822465e9b26 P 952dda47b0514e98a5fe41ff82b53748: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 03:26:34.991006 29585 tablet_bootstrap.cc:492] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42: No bootstrap required, opened a new log
I20260924 03:26:34.991174 29586 tablet_bootstrap.cc:654] T a9e04112036047bfb17de822465e9b26 P 3152be47624b416daf03651711f3fdc3: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:34.991250 29585 ts_tablet_manager.cc:1397] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260924 03:26:34.991875 29305 heartbeater.cc:503] Master 127.10.12.190:45999 was elected leader, sending a full tablet report...
I20260924 03:26:34.993235 29586 log.cc:824] T a9e04112036047bfb17de822465e9b26 P 3152be47624b416daf03651711f3fdc3: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:35.000519 29585 raft_consensus.cc:348] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3152be47624b416daf03651711f3fdc3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 34287 } } peers { permanent_uuid: "59284551533f47e9b56e5a1087371e42" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42647 } } peers { permanent_uuid: "952dda47b0514e98a5fe41ff82b53748" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 46155 } }
I20260924 03:26:35.001048 29585 raft_consensus.cc:374] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:35.001145 29585 raft_consensus.cc:729] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59284551533f47e9b56e5a1087371e42, State: Initialized, Role: FOLLOWER
I20260924 03:26:35.001776 29585 consensus_queue.cc:260] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3152be47624b416daf03651711f3fdc3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 34287 } } peers { permanent_uuid: "59284551533f47e9b56e5a1087371e42" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42647 } } peers { permanent_uuid: "952dda47b0514e98a5fe41ff82b53748" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 46155 } }
I20260924 03:26:35.001905 29584 raft_consensus.cc:348] T a9e04112036047bfb17de822465e9b26 P 952dda47b0514e98a5fe41ff82b53748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3152be47624b416daf03651711f3fdc3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 34287 } } peers { permanent_uuid: "59284551533f47e9b56e5a1087371e42" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42647 } } peers { permanent_uuid: "952dda47b0514e98a5fe41ff82b53748" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 46155 } }
I20260924 03:26:35.002362 29584 raft_consensus.cc:374] T a9e04112036047bfb17de822465e9b26 P 952dda47b0514e98a5fe41ff82b53748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:35.002470 29584 raft_consensus.cc:729] T a9e04112036047bfb17de822465e9b26 P 952dda47b0514e98a5fe41ff82b53748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 952dda47b0514e98a5fe41ff82b53748, State: Initialized, Role: FOLLOWER
I20260924 03:26:35.003122 29436 heartbeater.cc:503] Master 127.10.12.190:45999 was elected leader, sending a full tablet report...
I20260924 03:26:35.003090 29584 consensus_queue.cc:260] T a9e04112036047bfb17de822465e9b26 P 952dda47b0514e98a5fe41ff82b53748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3152be47624b416daf03651711f3fdc3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 34287 } } peers { permanent_uuid: "59284551533f47e9b56e5a1087371e42" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42647 } } peers { permanent_uuid: "952dda47b0514e98a5fe41ff82b53748" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 46155 } }
I20260924 03:26:35.003409 29586 tablet_bootstrap.cc:492] T a9e04112036047bfb17de822465e9b26 P 3152be47624b416daf03651711f3fdc3: No bootstrap required, opened a new log
I20260924 03:26:35.003666 29585 ts_tablet_manager.cc:1428] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42: Time spent starting tablet: real 0.012s user 0.013s sys 0.001s
I20260924 03:26:35.003722 29586 ts_tablet_manager.cc:1397] T a9e04112036047bfb17de822465e9b26 P 3152be47624b416daf03651711f3fdc3: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20260924 03:26:35.005300 29584 ts_tablet_manager.cc:1428] T a9e04112036047bfb17de822465e9b26 P 952dda47b0514e98a5fe41ff82b53748: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260924 03:26:35.012723 29586 raft_consensus.cc:348] T a9e04112036047bfb17de822465e9b26 P 3152be47624b416daf03651711f3fdc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3152be47624b416daf03651711f3fdc3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 34287 } } peers { permanent_uuid: "59284551533f47e9b56e5a1087371e42" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42647 } } peers { permanent_uuid: "952dda47b0514e98a5fe41ff82b53748" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 46155 } }
I20260924 03:26:35.013195 29586 raft_consensus.cc:374] T a9e04112036047bfb17de822465e9b26 P 3152be47624b416daf03651711f3fdc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:35.013295 29586 raft_consensus.cc:729] T a9e04112036047bfb17de822465e9b26 P 3152be47624b416daf03651711f3fdc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3152be47624b416daf03651711f3fdc3, State: Initialized, Role: FOLLOWER
I20260924 03:26:35.013890 29586 consensus_queue.cc:260] T a9e04112036047bfb17de822465e9b26 P 3152be47624b416daf03651711f3fdc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3152be47624b416daf03651711f3fdc3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 34287 } } peers { permanent_uuid: "59284551533f47e9b56e5a1087371e42" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42647 } } peers { permanent_uuid: "952dda47b0514e98a5fe41ff82b53748" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 46155 } }
I20260924 03:26:35.014957 29566 heartbeater.cc:503] Master 127.10.12.190:45999 was elected leader, sending a full tablet report...
I20260924 03:26:35.015614 29586 ts_tablet_manager.cc:1428] T a9e04112036047bfb17de822465e9b26 P 3152be47624b416daf03651711f3fdc3: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260924 03:26:35.074395 29590 raft_consensus.cc:482] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:26:35.074718 29590 raft_consensus.cc:504] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3152be47624b416daf03651711f3fdc3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 34287 } } peers { permanent_uuid: "59284551533f47e9b56e5a1087371e42" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42647 } } peers { permanent_uuid: "952dda47b0514e98a5fe41ff82b53748" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 46155 } }
I20260924 03:26:35.076695 29590 leader_election.cc:288] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3152be47624b416daf03651711f3fdc3 (127.10.12.131:34287), 952dda47b0514e98a5fe41ff82b53748 (127.10.12.129:46155)
I20260924 03:26:35.086164 29581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:35.077345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:36381 (local address 127.10.12.129:46155)
0924 03:26:35.077538 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:35.077544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:35.078203 (+ 659us) server_negotiation.cc:407] Connection header received
0924 03:26:35.078430 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:35.078435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:35.078528 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:35.078610 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:35.079383 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:35.080456 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:35.083663 (+ 3207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:35.085018 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:35.085179 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:35.085252 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:35.085747 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 03:26:35.085901 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 03:26:35.086164 29579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:35.077212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:39007 (local address 127.10.12.131:34287)
0924 03:26:35.077436 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:35.077442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:35.078150 (+ 708us) server_negotiation.cc:407] Connection header received
0924 03:26:35.078342 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:35.078347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:35.078432 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:35.078538 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:35.079271 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:35.080456 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:35.083679 (+ 3223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:35.084996 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:35.085184 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:35.085252 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:35.085747 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 03:26:35.085891 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 03:26:35.086604 29593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:35.077206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:46155 (local address 127.10.12.130:36381)
0924 03:26:35.078078 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:35.078091 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:35.078239 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:35.078637 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:35.078641 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:35.078660 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:35.079136 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:35.079146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:35.080625 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:35.080631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:35.083373 (+ 2742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:35.083400 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:35.085309 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:35.085316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:35.085492 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:35.085558 (+ 66us) client_negotiation.cc:769] Sending connection context
0924 03:26:35.085671 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 03:26:35.085884 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":206,"spinlock_wait_cycles":25472,"thread_start_us":365,"threads_started":1}
I20260924 03:26:35.086645 29594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:35.077209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:34287 (local address 127.10.12.130:39007)
0924 03:26:35.078001 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:35.078021 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:35.078198 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:35.078578 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:35.078583 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:35.078615 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:35.079078 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:35.079095 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:35.080625 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:35.080631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:35.083383 (+ 2752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:35.083400 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:35.085292 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:35.085300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:35.085572 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:35.085595 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 03:26:35.085671 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 03:26:35.085874 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":287,"thread_start_us":478,"threads_started":1}
I20260924 03:26:35.088074 29521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e04112036047bfb17de822465e9b26" candidate_uuid: "59284551533f47e9b56e5a1087371e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3152be47624b416daf03651711f3fdc3" is_pre_election: true
I20260924 03:26:35.088199 29260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e04112036047bfb17de822465e9b26" candidate_uuid: "59284551533f47e9b56e5a1087371e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "952dda47b0514e98a5fe41ff82b53748" is_pre_election: true
I20260924 03:26:35.088621 29521 raft_consensus.cc:2454] T a9e04112036047bfb17de822465e9b26 P 3152be47624b416daf03651711f3fdc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59284551533f47e9b56e5a1087371e42 in term 0.
I20260924 03:26:35.088936 29260 raft_consensus.cc:2454] T a9e04112036047bfb17de822465e9b26 P 952dda47b0514e98a5fe41ff82b53748 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59284551533f47e9b56e5a1087371e42 in term 0.
I20260924 03:26:35.089357 29325 leader_election.cc:302] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3152be47624b416daf03651711f3fdc3, 59284551533f47e9b56e5a1087371e42; no voters:
I20260924 03:26:35.089869 29590 raft_consensus.cc:2781] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:26:35.090032 29590 raft_consensus.cc:482] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:26:35.090137 29590 raft_consensus.cc:3037] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:35.092433 29590 raft_consensus.cc:504] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3152be47624b416daf03651711f3fdc3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 34287 } } peers { permanent_uuid: "59284551533f47e9b56e5a1087371e42" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42647 } } peers { permanent_uuid: "952dda47b0514e98a5fe41ff82b53748" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 46155 } }
I20260924 03:26:35.093091 29590 leader_election.cc:288] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42 [CANDIDATE]: Term 1 election: Requested vote from peers 3152be47624b416daf03651711f3fdc3 (127.10.12.131:34287), 952dda47b0514e98a5fe41ff82b53748 (127.10.12.129:46155)
I20260924 03:26:35.093663 29521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e04112036047bfb17de822465e9b26" candidate_uuid: "59284551533f47e9b56e5a1087371e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3152be47624b416daf03651711f3fdc3"
I20260924 03:26:35.093745 29260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e04112036047bfb17de822465e9b26" candidate_uuid: "59284551533f47e9b56e5a1087371e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "952dda47b0514e98a5fe41ff82b53748"
I20260924 03:26:35.093876 29521 raft_consensus.cc:3037] T a9e04112036047bfb17de822465e9b26 P 3152be47624b416daf03651711f3fdc3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:35.093972 29260 raft_consensus.cc:3037] T a9e04112036047bfb17de822465e9b26 P 952dda47b0514e98a5fe41ff82b53748 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:35.096284 29521 raft_consensus.cc:2454] T a9e04112036047bfb17de822465e9b26 P 3152be47624b416daf03651711f3fdc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59284551533f47e9b56e5a1087371e42 in term 1.
I20260924 03:26:35.096338 29260 raft_consensus.cc:2454] T a9e04112036047bfb17de822465e9b26 P 952dda47b0514e98a5fe41ff82b53748 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59284551533f47e9b56e5a1087371e42 in term 1.
I20260924 03:26:35.096812 29325 leader_election.cc:302] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3152be47624b416daf03651711f3fdc3, 59284551533f47e9b56e5a1087371e42, 952dda47b0514e98a5fe41ff82b53748; no voters:
I20260924 03:26:35.097235 29590 raft_consensus.cc:2781] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:26:35.097711 29590 raft_consensus.cc:686] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42 [term 1 LEADER]: Becoming Leader. State: Replica: 59284551533f47e9b56e5a1087371e42, State: Running, Role: LEADER
I20260924 03:26:35.098263 29590 consensus_queue.cc:237] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3152be47624b416daf03651711f3fdc3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 34287 } } peers { permanent_uuid: "59284551533f47e9b56e5a1087371e42" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42647 } } peers { permanent_uuid: "952dda47b0514e98a5fe41ff82b53748" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 46155 } }
I20260924 03:26:35.103081 29118 catalog_manager.cc:5495] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59284551533f47e9b56e5a1087371e42 (127.10.12.130). New cstate: current_term: 1 leader_uuid: "59284551533f47e9b56e5a1087371e42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3152be47624b416daf03651711f3fdc3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 34287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59284551533f47e9b56e5a1087371e42" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 42647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "952dda47b0514e98a5fe41ff82b53748" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 46155 } health_report { overall_health: UNKNOWN } } }
W20260924 03:26:35.128741 29567 tablet.cc:2369] T a9e04112036047bfb17de822465e9b26 P 3152be47624b416daf03651711f3fdc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:26:35.155287 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:35.149446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58808 (local address 127.10.12.130:42647)
0924 03:26:35.149617 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:35.149621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:35.149713 (+ 92us) server_negotiation.cc:407] Connection header received
0924 03:26:35.149962 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:35.149967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:35.150085 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:35.150222 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:26:35.150736 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:35.151579 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:35.153145 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:35.153616 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:35.153718 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:35.154243 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:26:35.154757 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:26:35.154908 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:35.155027 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 03:26:35.155068 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 03:26:35.164376 29260 raft_consensus.cc:1261] T a9e04112036047bfb17de822465e9b26 P 952dda47b0514e98a5fe41ff82b53748 [term 1 FOLLOWER]: Refusing update from remote peer 59284551533f47e9b56e5a1087371e42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:26:35.164397 29521 raft_consensus.cc:1261] T a9e04112036047bfb17de822465e9b26 P 3152be47624b416daf03651711f3fdc3 [term 1 FOLLOWER]: Refusing update from remote peer 59284551533f47e9b56e5a1087371e42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:26:35.165405 29590 consensus_queue.cc:1035] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3152be47624b416daf03651711f3fdc3" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 34287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:26:35.165735 29595 consensus_queue.cc:1035] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952dda47b0514e98a5fe41ff82b53748" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 46155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 03:26:35.171633 29437 tablet.cc:2369] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:26:35.178076 29598 mvcc.cc:205] Tried to move back new op lower bound from 7332742738582532096 to 7332742738326626304. Current Snapshot: MvccSnapshot[applied={T|T < 7332742738582532096}]
I20260924 03:26:35.178542 29599 mvcc.cc:205] Tried to move back new op lower bound from 7332742738582532096 to 7332742738326626304. Current Snapshot: MvccSnapshot[applied={T|T < 7332742738582532096}]
I20260924 03:26:35.182883 29600 mvcc.cc:205] Tried to move back new op lower bound from 7332742738582532096 to 7332742738326626304. Current Snapshot: MvccSnapshot[applied={T|T < 7332742738582532096}]
I20260924 03:26:35.193966 29118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:26:35.194294 29118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:26:35.198556 29118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f95701f3d13444499426a91341a01b79: Sending DeleteTablet for 3 replicas of tablet a9e04112036047bfb17de822465e9b26
I20260924 03:26:35.200021 29501 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e04112036047bfb17de822465e9b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:26:35 UTC) from {username='slave'} at 127.0.0.1:41276
I20260924 03:26:35.200070 29371 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e04112036047bfb17de822465e9b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:26:35 UTC) from {username='slave'} at 127.0.0.1:58794
I20260924 03:26:35.200070 29240 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e04112036047bfb17de822465e9b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:26:35 UTC) from {username='slave'} at 127.0.0.1:53450
I20260924 03:26:35.203505 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 29178
I20260924 03:26:35.220791 29609 tablet_replica.cc:330] T a9e04112036047bfb17de822465e9b26 P 952dda47b0514e98a5fe41ff82b53748: stopping tablet replica
I20260924 03:26:35.221455 29609 raft_consensus.cc:2229] T a9e04112036047bfb17de822465e9b26 P 952dda47b0514e98a5fe41ff82b53748 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:26:35.222074 29609 raft_consensus.cc:2258] T a9e04112036047bfb17de822465e9b26 P 952dda47b0514e98a5fe41ff82b53748 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:26:35.222983 29610 tablet_replica.cc:330] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42: stopping tablet replica
I20260924 03:26:35.223811 29608 tablet_replica.cc:330] T a9e04112036047bfb17de822465e9b26 P 3152be47624b416daf03651711f3fdc3: stopping tablet replica
I20260924 03:26:35.224339 29608 raft_consensus.cc:2229] T a9e04112036047bfb17de822465e9b26 P 3152be47624b416daf03651711f3fdc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:26:35.224952 29608 raft_consensus.cc:2258] T a9e04112036047bfb17de822465e9b26 P 3152be47624b416daf03651711f3fdc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:26:35.225260 29581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:35.205822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.10.12.129:46155)
0924 03:26:35.206029 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:35.206035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:35.209766 (+ 3731us) server_negotiation.cc:407] Connection header received
0924 03:26:35.210723 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:35.210730 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:35.210874 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:35.210955 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:35.212216 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:35.213549 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:35.217664 (+ 4115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:35.218419 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:35.218534 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:35.223422 (+ 4888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:35.223435 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:35.223455 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:35.224301 (+ 846us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:35.224385 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:35.224414 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:35.224536 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:35.224770 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 03:26:35.224909 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:26:35.225652 29609 ts_tablet_manager.cc:1905] T a9e04112036047bfb17de822465e9b26 P 952dda47b0514e98a5fe41ff82b53748: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:26:35.226027 29610 raft_consensus.cc:2229] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:26:35.226797 29300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:35.226826 29610 raft_consensus.cc:2258] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:26:35.228919 29608 ts_tablet_manager.cc:1905] T a9e04112036047bfb17de822465e9b26 P 3152be47624b416daf03651711f3fdc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:26:35.229648 29610 ts_tablet_manager.cc:1905] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:26:35.237131 29608 ts_tablet_manager.cc:1918] T a9e04112036047bfb17de822465e9b26 P 3152be47624b416daf03651711f3fdc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:26:35.237428 29608 log.cc:1196] T a9e04112036047bfb17de822465e9b26 P 3152be47624b416daf03651711f3fdc3: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/a9e04112036047bfb17de822465e9b26
I20260924 03:26:35.238230 29608 ts_tablet_manager.cc:1939] T a9e04112036047bfb17de822465e9b26 P 3152be47624b416daf03651711f3fdc3: Deleting consensus metadata
I20260924 03:26:35.238700 29610 ts_tablet_manager.cc:1918] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:26:35.239001 29610 log.cc:1196] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/a9e04112036047bfb17de822465e9b26
I20260924 03:26:35.239603 29610 ts_tablet_manager.cc:1939] T a9e04112036047bfb17de822465e9b26 P 59284551533f47e9b56e5a1087371e42: Deleting consensus metadata
I20260924 03:26:35.240593 29104 catalog_manager.cc:4841] TS 3152be47624b416daf03651711f3fdc3 (127.10.12.131:34287): tablet a9e04112036047bfb17de822465e9b26 (table test-table [id=e416e06a2c9c45d693195abd4fdfcdd2]) successfully deleted
I20260924 03:26:35.241438 29104 catalog_manager.cc:4841] TS 59284551533f47e9b56e5a1087371e42 (127.10.12.130:42647): tablet a9e04112036047bfb17de822465e9b26 (table test-table [id=e416e06a2c9c45d693195abd4fdfcdd2]) successfully deleted
I20260924 03:26:35.340731 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 29178
I20260924 03:26:35.355404 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 29309
W20260924 03:26:35.355445 29105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.12.129:46155 (error 108)
I20260924 03:26:35.357376 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:35.356862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:46155 (local address 127.0.0.1:53464)
0924 03:26:35.357058 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:35.357197 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 03:26:35.357642 29105 catalog_manager.cc:4584] TS 952dda47b0514e98a5fe41ff82b53748 (127.10.12.129:46155): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9e04112036047bfb17de822465e9b26: Network error: Client connection negotiation failed: client connection to 127.10.12.129:46155: connect: Connection refused (error 111)
I20260924 03:26:35.363936 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:35.356222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58818 (local address 127.10.12.130:42647)
0924 03:26:35.356429 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:35.356435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:35.356723 (+ 288us) server_negotiation.cc:407] Connection header received
0924 03:26:35.356800 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:35.356810 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:35.356942 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:35.357047 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:35.357612 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:35.358527 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:35.360632 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:35.361135 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:35.361246 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:35.362447 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:35.362458 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:35.362473 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:35.363129 (+ 656us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:35.363180 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:35.363202 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:35.363299 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:35.363453 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 03:26:35.363544 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 03:26:35.364718 29431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:35.411382 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:35.410827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:46155 (local address 127.0.0.1:53470)
0924 03:26:35.411075 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:35.411185 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260924 03:26:35.412565 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:35.412156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:46155 (local address 127.0.0.1:53486)
0924 03:26:35.412319 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:35.412423 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 03:26:35.473531 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:35.472993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:46155 (local address 127.0.0.1:53494)
0924 03:26:35.473252 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:35.473355 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260924 03:26:35.474659 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:35.474289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:46155 (local address 127.0.0.1:53510)
0924 03:26:35.474455 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:35.474530 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 03:26:35.481844 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 29309
I20260924 03:26:35.497653 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 29439
I20260924 03:26:35.504972 29579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:35.498707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41278 (local address 127.10.12.131:34287)
0924 03:26:35.498969 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:35.498976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:35.499057 (+ 81us) server_negotiation.cc:407] Connection header received
0924 03:26:35.499253 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:35.499257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:35.499345 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:35.499447 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:35.499880 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:35.500645 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:35.501952 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:35.502341 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:35.502455 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:35.503523 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:35.503532 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:35.503547 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:35.504220 (+ 673us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:35.504274 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:35.504297 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:35.504411 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:35.504534 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 03:26:35.504653 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260924 03:26:35.505736 29561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:35.555430 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:35.554948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:46155 (local address 127.0.0.1:53526)
0924 03:26:35.555162 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:35.555254 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 03:26:35.556592 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:35.556167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:46155 (local address 127.0.0.1:53532)
0924 03:26:35.556379 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:35.556465 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:46155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 03:26:35.595396 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 29439
I20260924 03:26:35.608232 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 29088
I20260924 03:26:35.609426 29149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:35.720584 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 29088
2026-09-24T03:26:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2876 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:26:35 dist-test-slave-00qs krb5kdc[29619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:26:35 dist-test-slave-00qs krb5kdc[29619](info): set up 2 sockets
Sep 24 03:26:35 dist-test-slave-00qs krb5kdc[29619](info): commencing operation
krb5kdc: starting...
W20260924 03:26:36.773016 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:26:36 dist-test-slave-00qs krb5kdc[29619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:26:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:26:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:26:37.097796 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:36925
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:36925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:26:37.555193 29632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:37.555613 29632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:37.555780 29632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:37.566390 29632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:26:37.566618 29632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:37.566715 29632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:37.566815 29632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:26:37.566902 29632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:26:37.579689 29632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:36925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:36925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:37.581650 29632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:37.583343 29632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:37.599289 29637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:37.599208 29638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:37.599181 29640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:37.600965 29632 server_base.cc:1034] running on GCE node
I20260924 03:26:37.602205 29632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:37.604912 29632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:37.606249 29632 hybrid_clock.cc:648] HybridClock initialized: now 1790220397606152 us; error 42 us; skew 500 ppm
I20260924 03:26:37.606775 29632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:26:37 dist-test-slave-00qs krb5kdc[29619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220397, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:26:37.610929 29632 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:26:37.613086 29632 webserver.cc:460] Webserver started at http://127.10.12.190:34693/ using document root <none> and password file <none>
I20260924 03:26:37.614123 29632 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:37.614305 29632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:37.614773 29632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:37.618781 29632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "010a6de3e32f487fb5557b0b874aca27"
format_stamp: "Formatted at 2026-09-24 03:26:37 on dist-test-slave-00qs"
server_key: "f067257be92a00c76d591fefb1403ef2"
server_key_iv: "81ed8809ca36c6010f3a15e730b79773"
server_key_version: "encryptionkey@0"
I20260924 03:26:37.619764 29632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "010a6de3e32f487fb5557b0b874aca27"
format_stamp: "Formatted at 2026-09-24 03:26:37 on dist-test-slave-00qs"
server_key: "f067257be92a00c76d591fefb1403ef2"
server_key_iv: "81ed8809ca36c6010f3a15e730b79773"
server_key_version: "encryptionkey@0"
I20260924 03:26:37.626511 29632 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 03:26:37.631333 29647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:37.633265 29632 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 03:26:37.633518 29632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "010a6de3e32f487fb5557b0b874aca27"
format_stamp: "Formatted at 2026-09-24 03:26:37 on dist-test-slave-00qs"
server_key: "f067257be92a00c76d591fefb1403ef2"
server_key_iv: "81ed8809ca36c6010f3a15e730b79773"
server_key_version: "encryptionkey@0"
I20260924 03:26:37.633797 29632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:37.659649 29632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:37.665143 29632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:37.665477 29632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:37.686455 29632 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:36925
I20260924 03:26:37.686460 29698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:36925 every 8 connection(s)
I20260924 03:26:37.688292 29632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:26:37.694067 29699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:37.696663 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 29632
I20260924 03:26:37.696931 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:26:37.697580 10290 external_mini_cluster.cc:1428] Setting key da4d0f51c3002aed477335c59b6a14d8
I20260924 03:26:37.709160 29699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 010a6de3e32f487fb5557b0b874aca27: Bootstrap starting.
I20260924 03:26:37.713276 29699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 010a6de3e32f487fb5557b0b874aca27: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:37.714643 29699 log.cc:824] T 00000000000000000000000000000000 P 010a6de3e32f487fb5557b0b874aca27: Log is configured to *not* fsync() on all Append() calls
Sep 24 03:26:37 dist-test-slave-00qs krb5kdc[29619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:26:37.719266 29699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 010a6de3e32f487fb5557b0b874aca27: No bootstrap required, opened a new log
I20260924 03:26:37.727265 29702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:37.700537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47296 (local address 127.10.12.190:36925)
0924 03:26:37.701694 (+ 1157us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:37.701717 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:37.701781 (+ 64us) server_negotiation.cc:407] Connection header received
0924 03:26:37.703927 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:37.704000 (+ 73us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:37.704625 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:37.705169 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:37.707251 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:37.708976 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:37.710489 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:37.711319 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:37.711538 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:37.716604 (+ 5066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:37.716617 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:37.716646 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:37.716716 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:37.721134 (+ 4418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:37.722152 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:37.722163 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:37.722176 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:37.722454 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:37.722964 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:37.722970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:37.722974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:37.723905 (+ 931us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:37.724292 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:37.724791 (+ 499us) server_negotiation.cc:299] Negotiation successful
0924 03:26:37.725423 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":818,"thread_start_us":465,"threads_started":1}
I20260924 03:26:37.730928 29699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 010a6de3e32f487fb5557b0b874aca27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "010a6de3e32f487fb5557b0b874aca27" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36925 } }
I20260924 03:26:37.731520 29699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 010a6de3e32f487fb5557b0b874aca27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:37.731647 29699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 010a6de3e32f487fb5557b0b874aca27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 010a6de3e32f487fb5557b0b874aca27, State: Initialized, Role: FOLLOWER
I20260924 03:26:37.732389 29699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 010a6de3e32f487fb5557b0b874aca27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "010a6de3e32f487fb5557b0b874aca27" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36925 } }
I20260924 03:26:37.732671 29699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 010a6de3e32f487fb5557b0b874aca27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:26:37.732781 29699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 010a6de3e32f487fb5557b0b874aca27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:26:37.732928 29699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 010a6de3e32f487fb5557b0b874aca27 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:37.738374 29699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 010a6de3e32f487fb5557b0b874aca27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "010a6de3e32f487fb5557b0b874aca27" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36925 } }
I20260924 03:26:37.739187 29699 leader_election.cc:302] T 00000000000000000000000000000000 P 010a6de3e32f487fb5557b0b874aca27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 010a6de3e32f487fb5557b0b874aca27; no voters:
I20260924 03:26:37.739904 29699 leader_election.cc:288] T 00000000000000000000000000000000 P 010a6de3e32f487fb5557b0b874aca27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:26:37.740085 29704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 010a6de3e32f487fb5557b0b874aca27 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:26:37.740801 29704 raft_consensus.cc:686] T 00000000000000000000000000000000 P 010a6de3e32f487fb5557b0b874aca27 [term 1 LEADER]: Becoming Leader. State: Replica: 010a6de3e32f487fb5557b0b874aca27, State: Running, Role: LEADER
I20260924 03:26:37.741457 29704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 010a6de3e32f487fb5557b0b874aca27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "010a6de3e32f487fb5557b0b874aca27" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36925 } }
I20260924 03:26:37.742394 29699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 010a6de3e32f487fb5557b0b874aca27 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:26:37.744726 29705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 010a6de3e32f487fb5557b0b874aca27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "010a6de3e32f487fb5557b0b874aca27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "010a6de3e32f487fb5557b0b874aca27" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36925 } } }
I20260924 03:26:37.745213 29705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 010a6de3e32f487fb5557b0b874aca27 [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:37.745954 29706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 010a6de3e32f487fb5557b0b874aca27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 010a6de3e32f487fb5557b0b874aca27. Latest consensus state: current_term: 1 leader_uuid: "010a6de3e32f487fb5557b0b874aca27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "010a6de3e32f487fb5557b0b874aca27" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36925 } } }
I20260924 03:26:37.746305 29706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 010a6de3e32f487fb5557b0b874aca27 [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:37.746232 29710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:26:37.753643 29710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:26:37.766162 29710 catalog_manager.cc:1347] Generated new cluster ID: 76a6a4326b7f4187b9214fd7edf4833c
I20260924 03:26:37.766350 29710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:26:37.790553 29710 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:26:37.792989 29710 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:26:37.806921 29710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 010a6de3e32f487fb5557b0b874aca27: Generated new TSK 0
I20260924 03:26:37.808525 29710 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
Entry for principal kudu/127.10.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab.
I20260924 03:26:37.901016 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:36925
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:26:38.349252 29725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:38.349712 29725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:38.349877 29725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:38.358347 29725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:38.358510 29725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:38.358682 29725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:26:38.370247 29725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.129.keytab
--principal=kudu/127.10.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:36925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:38.372138 29725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:38.373939 29725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:38.391137 29733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:38.391067 29731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:38.391099 29730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:38.392190 29725 server_base.cc:1034] running on GCE node
I20260924 03:26:38.393010 29725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:38.394265 29725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:38.395489 29725 hybrid_clock.cc:648] HybridClock initialized: now 1790220398395428 us; error 53 us; skew 500 ppm
I20260924 03:26:38.395874 29725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:26:38 dist-test-slave-00qs krb5kdc[29619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220398, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:26:38.399827 29725 init.cc:377] Logged in from keytab as kudu/127.10.12.129@KRBTEST.COM (short username kudu)
I20260924 03:26:38.401752 29725 webserver.cc:460] Webserver started at http://127.10.12.129:44455/ using document root <none> and password file <none>
I20260924 03:26:38.402674 29725 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:38.402830 29725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:38.403210 29725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:38.407032 29725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "178242d6473e4e13afe67a3dab206f4b"
format_stamp: "Formatted at 2026-09-24 03:26:38 on dist-test-slave-00qs"
server_key: "02fb017ed22035e53830441e860f1339"
server_key_iv: "21e17f889e3aa90b8dd93a4b06ed6218"
server_key_version: "encryptionkey@0"
I20260924 03:26:38.408104 29725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "178242d6473e4e13afe67a3dab206f4b"
format_stamp: "Formatted at 2026-09-24 03:26:38 on dist-test-slave-00qs"
server_key: "02fb017ed22035e53830441e860f1339"
server_key_iv: "21e17f889e3aa90b8dd93a4b06ed6218"
server_key_version: "encryptionkey@0"
I20260924 03:26:38.415045 29725 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260924 03:26:38.419488 29740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:38.421419 29725 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260924 03:26:38.421617 29725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "178242d6473e4e13afe67a3dab206f4b"
format_stamp: "Formatted at 2026-09-24 03:26:38 on dist-test-slave-00qs"
server_key: "02fb017ed22035e53830441e860f1339"
server_key_iv: "21e17f889e3aa90b8dd93a4b06ed6218"
server_key_version: "encryptionkey@0"
I20260924 03:26:38.421872 29725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:38.443651 29725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:38.449390 29725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:38.449733 29725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:38.450961 29725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:26:38.452724 29725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:26:38.452813 29725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:38.452955 29725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:26:38.453011 29725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:38.490151 29725 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:35741
I20260924 03:26:38.490206 29852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:35741 every 8 connection(s)
I20260924 03:26:38.491784 29725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:26:38.497540 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 29725
I20260924 03:26:38.497792 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:26:38.498401 10290 external_mini_cluster.cc:1428] Setting key 28d12b54f80a1fcf121a6e34ac253913
Sep 24 03:26:38 dist-test-slave-00qs krb5kdc[29619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220398, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.129@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:26:38.517432 29855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:38.495182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:34539 (local address 127.10.12.190:36925)
0924 03:26:38.495743 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:38.495750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:38.497085 (+ 1335us) server_negotiation.cc:407] Connection header received
0924 03:26:38.499810 (+ 2725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:38.499815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:38.499912 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:38.500055 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:38.502257 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:38.503068 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:38.504895 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:38.505336 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:38.505457 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:38.509921 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:38.509929 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:38.509934 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:38.509987 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:38.513066 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:38.514270 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:38.514275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:38.514277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:38.514445 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:38.515053 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:38.515057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:38.515059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:38.515423 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:38.515558 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:38.516817 (+ 1259us) server_negotiation.cc:299] Negotiation successful
0924 03:26:38.517103 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":285,"threads_started":1}
I20260924 03:26:38.519153 29856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:38.495893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36925 (local address 127.10.12.129:34539)
0924 03:26:38.496863 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:38.496916 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:38.499494 (+ 2578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:38.500544 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:38.500560 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:38.501302 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:26:38.502048 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:38.502070 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:38.503244 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:38.503248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:38.504736 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:38.504751 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:38.505596 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:38.505603 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:38.505871 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:38.506695 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:26:38.506727 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:26:38.509508 (+ 2781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:26:38.513255 (+ 3747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:26:38.513267 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:26:38.513291 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:26:38.514083 (+ 792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:26:38.514557 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:26:38.514562 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:26:38.514565 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:26:38.514907 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:26:38.515554 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:26:38.515564 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:26:38.516218 (+ 654us) client_negotiation.cc:769] Sending connection context
0924 03:26:38.516576 (+ 358us) client_negotiation.cc:240] Negotiation successful
0924 03:26:38.517285 (+ 709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":355,"threads_started":1}
I20260924 03:26:38.521253 29853 heartbeater.cc:347] Connected to a master server at 127.10.12.190:36925
I20260924 03:26:38.521618 29853 heartbeater.cc:464] Registering TS with master...
I20260924 03:26:38.522603 29853 heartbeater.cc:511] Master 127.10.12.190:36925 requested a full tablet report, sending...
I20260924 03:26:38.525596 29664 ts_manager.cc:194] Registered new tserver with Master: 178242d6473e4e13afe67a3dab206f4b (127.10.12.129:35741)
I20260924 03:26:38.527603 29664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.129@KRBTEST.COM'} at 127.10.12.129:34539
WARNING: no policy specified for kudu/127.10.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
Entry for principal kudu/127.10.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab.
I20260924 03:26:38.569295 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:36925
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:26:38.975587 29859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:38.976081 29859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:38.976241 29859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:38.984501 29859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:38.984680 29859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:38.984846 29859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:26:38.997202 29859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.130.keytab
--principal=kudu/127.10.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:36925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:38.999322 29859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:39.001318 29859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:39.018224 29867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:39.018481 29864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:39.018209 29865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:39.020159 29859 server_base.cc:1034] running on GCE node
I20260924 03:26:39.021097 29859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:39.022367 29859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:39.023645 29859 hybrid_clock.cc:648] HybridClock initialized: now 1790220399023567 us; error 59 us; skew 500 ppm
I20260924 03:26:39.024115 29859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:26:39 dist-test-slave-00qs krb5kdc[29619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220399, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:26:39.028498 29859 init.cc:377] Logged in from keytab as kudu/127.10.12.130@KRBTEST.COM (short username kudu)
I20260924 03:26:39.030813 29859 webserver.cc:460] Webserver started at http://127.10.12.130:39911/ using document root <none> and password file <none>
I20260924 03:26:39.031839 29859 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:39.032119 29859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:39.032745 29859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:39.037310 29859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "f850a85381b8448181438dd7d44d75a7"
format_stamp: "Formatted at 2026-09-24 03:26:39 on dist-test-slave-00qs"
server_key: "f6c9f6dc2af62a76d9bf4661dc749441"
server_key_iv: "fea5c77b88a91037a9e7022e0e93ca76"
server_key_version: "encryptionkey@0"
I20260924 03:26:39.038988 29859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "f850a85381b8448181438dd7d44d75a7"
format_stamp: "Formatted at 2026-09-24 03:26:39 on dist-test-slave-00qs"
server_key: "f6c9f6dc2af62a76d9bf4661dc749441"
server_key_iv: "fea5c77b88a91037a9e7022e0e93ca76"
server_key_version: "encryptionkey@0"
I20260924 03:26:39.046767 29859 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260924 03:26:39.051771 29874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:39.053679 29859 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:26:39.053905 29859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "f850a85381b8448181438dd7d44d75a7"
format_stamp: "Formatted at 2026-09-24 03:26:39 on dist-test-slave-00qs"
server_key: "f6c9f6dc2af62a76d9bf4661dc749441"
server_key_iv: "fea5c77b88a91037a9e7022e0e93ca76"
server_key_version: "encryptionkey@0"
I20260924 03:26:39.054195 29859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:39.070698 29859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:39.075914 29859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:39.076318 29859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:39.077567 29859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:26:39.079353 29859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:26:39.079442 29859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:39.079586 29859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:26:39.079644 29859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:39.116261 29859 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:44971
I20260924 03:26:39.116300 29986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:44971 every 8 connection(s)
I20260924 03:26:39.117794 29859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:26:39.123137 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 29859
I20260924 03:26:39.123382 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:26:39.124035 10290 external_mini_cluster.cc:1428] Setting key dce3dcf600dc005cf3956c4bf65ebe6b
Sep 24 03:26:39 dist-test-slave-00qs krb5kdc[29619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220399, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.130@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:26:39.144784 29989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:39.121034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:57587 (local address 127.10.12.190:36925)
0924 03:26:39.121609 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:39.121616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:39.122916 (+ 1300us) server_negotiation.cc:407] Connection header received
0924 03:26:39.125758 (+ 2842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:39.125763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:39.125859 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:39.125999 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:39.128459 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:39.129296 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:39.131677 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:39.132266 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:39.132387 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:39.137613 (+ 5226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:39.137620 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:39.137625 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:39.137679 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:39.140777 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:39.141784 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:39.141789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:39.141792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:39.141954 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:39.142515 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:39.142519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:39.142521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:39.142883 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:39.143033 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:39.144061 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0924 03:26:39.144378 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":276,"threads_started":1}
I20260924 03:26:39.145936 29990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:39.121734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36925 (local address 127.10.12.130:57587)
0924 03:26:39.122604 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:39.122664 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:39.125352 (+ 2688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:39.126255 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:39.126267 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:39.127176 (+ 909us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:26:39.128197 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:39.128227 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:39.129478 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:39.129485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:39.131474 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:39.131494 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:39.132533 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:39.132539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:39.132882 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:39.133801 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:26:39.133843 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:26:39.137227 (+ 3384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:26:39.141007 (+ 3780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:26:39.141017 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:26:39.141038 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:26:39.141599 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:26:39.142085 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:26:39.142089 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:26:39.142092 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:26:39.142390 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:26:39.143021 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:26:39.143032 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:26:39.143545 (+ 513us) client_negotiation.cc:769] Sending connection context
0924 03:26:39.143884 (+ 339us) client_negotiation.cc:240] Negotiation successful
0924 03:26:39.144477 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":324,"threads_started":1}
I20260924 03:26:39.147770 29987 heartbeater.cc:347] Connected to a master server at 127.10.12.190:36925
I20260924 03:26:39.148164 29987 heartbeater.cc:464] Registering TS with master...
I20260924 03:26:39.149139 29987 heartbeater.cc:511] Master 127.10.12.190:36925 requested a full tablet report, sending...
I20260924 03:26:39.151079 29664 ts_manager.cc:194] Registered new tserver with Master: f850a85381b8448181438dd7d44d75a7 (127.10.12.130:44971)
I20260924 03:26:39.152114 29664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.130@KRBTEST.COM'} at 127.10.12.130:57587
WARNING: no policy specified for kudu/127.10.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
Entry for principal kudu/127.10.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab.
I20260924 03:26:39.197273 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:36925
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:26:39.532037 29853 heartbeater.cc:503] Master 127.10.12.190:36925 was elected leader, sending a full tablet report...
W20260924 03:26:39.611238 29993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:39.611644 29993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:39.611778 29993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:39.620601 29993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:39.620746 29993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:39.620903 29993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:26:39.632306 29993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.131.keytab
--principal=kudu/127.10.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:36925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:39.634119 29993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:39.635789 29993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:39.652705 29998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:39.652601 29999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:39.652638 30001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:39.653538 29993 server_base.cc:1034] running on GCE node
I20260924 03:26:39.654368 29993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:39.655551 29993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:39.656810 29993 hybrid_clock.cc:648] HybridClock initialized: now 1790220399656734 us; error 64 us; skew 500 ppm
I20260924 03:26:39.657184 29993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:26:39 dist-test-slave-00qs krb5kdc[29619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220399, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:26:39.661216 29993 init.cc:377] Logged in from keytab as kudu/127.10.12.131@KRBTEST.COM (short username kudu)
I20260924 03:26:39.663093 29993 webserver.cc:460] Webserver started at http://127.10.12.131:45855/ using document root <none> and password file <none>
I20260924 03:26:39.663990 29993 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:39.664134 29993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:39.664494 29993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:39.668105 29993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "81e88a74da3e4ebd8be09bf82e605539"
format_stamp: "Formatted at 2026-09-24 03:26:39 on dist-test-slave-00qs"
server_key: "f188eeae08dae4a22242fd5ae64855c4"
server_key_iv: "d6f0433a565131235a2927624203fc9f"
server_key_version: "encryptionkey@0"
I20260924 03:26:39.669020 29993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "81e88a74da3e4ebd8be09bf82e605539"
format_stamp: "Formatted at 2026-09-24 03:26:39 on dist-test-slave-00qs"
server_key: "f188eeae08dae4a22242fd5ae64855c4"
server_key_iv: "d6f0433a565131235a2927624203fc9f"
server_key_version: "encryptionkey@0"
I20260924 03:26:39.675710 29993 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260924 03:26:39.680356 30008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:39.682171 29993 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:26:39.682379 29993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "81e88a74da3e4ebd8be09bf82e605539"
format_stamp: "Formatted at 2026-09-24 03:26:39 on dist-test-slave-00qs"
server_key: "f188eeae08dae4a22242fd5ae64855c4"
server_key_iv: "d6f0433a565131235a2927624203fc9f"
server_key_version: "encryptionkey@0"
I20260924 03:26:39.682628 29993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:39.722854 29993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:39.728521 29993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:39.728911 29993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:39.730111 29993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:26:39.731832 29993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:26:39.731920 29993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:39.732097 29993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:26:39.732156 29993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:39.769062 29993 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:36677
I20260924 03:26:39.769125 30120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:36677 every 8 connection(s)
I20260924 03:26:39.770681 29993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:26:39.772167 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 29993
I20260924 03:26:39.772372 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:26:39.772890 10290 external_mini_cluster.cc:1428] Setting key dba2c48422f0ce880868d770cc627fee
Sep 24 03:26:39 dist-test-slave-00qs krb5kdc[29619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220399, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.131@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:26:39.797216 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:39.773948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:50271 (local address 127.10.12.190:36925)
0924 03:26:39.774617 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:39.774624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:39.776265 (+ 1641us) server_negotiation.cc:407] Connection header received
0924 03:26:39.778672 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:39.778678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:39.778770 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:39.778879 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:39.781002 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:39.781957 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:39.783755 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:39.784299 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:39.784420 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:39.789789 (+ 5369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:39.789795 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:39.789799 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:39.789846 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:39.793098 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:39.794237 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:39.794243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:39.794246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:39.794406 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:39.795010 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:39.795015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:39.795017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:39.795402 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:39.795548 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:39.796620 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0924 03:26:39.796876 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":297,"threads_started":1}
I20260924 03:26:39.798534 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:39.774928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36925 (local address 127.10.12.131:50271)
0924 03:26:39.776058 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:39.776120 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:39.778263 (+ 2143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:39.779184 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:39.779197 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:39.779909 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:26:39.780720 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:39.780742 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:39.782133 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:39.782137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:39.783580 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:39.783596 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:39.784584 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:39.784590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:39.784895 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:39.785811 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:26:39.785849 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:26:39.789482 (+ 3633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:26:39.793368 (+ 3886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:26:39.793380 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:26:39.793402 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:26:39.794053 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:26:39.794540 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:26:39.794544 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:26:39.794547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:26:39.794852 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:26:39.795540 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:26:39.795549 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:26:39.796138 (+ 589us) client_negotiation.cc:769] Sending connection context
0924 03:26:39.796460 (+ 322us) client_negotiation.cc:240] Negotiation successful
0924 03:26:39.797024 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":424,"threads_started":1}
I20260924 03:26:39.800380 30121 heartbeater.cc:347] Connected to a master server at 127.10.12.190:36925
I20260924 03:26:39.800791 30121 heartbeater.cc:464] Registering TS with master...
I20260924 03:26:39.801916 30121 heartbeater.cc:511] Master 127.10.12.190:36925 requested a full tablet report, sending...
I20260924 03:26:39.803992 29664 ts_manager.cc:194] Registered new tserver with Master: 81e88a74da3e4ebd8be09bf82e605539 (127.10.12.131:36677)
I20260924 03:26:39.805013 29664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.12.131@KRBTEST.COM'} at 127.10.12.131:50271
I20260924 03:26:39.817273 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:26:39.838083 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:39.822271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58324 (local address 127.10.12.190:36925)
0924 03:26:39.822550 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:39.822557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:39.822892 (+ 335us) server_negotiation.cc:407] Connection header received
0924 03:26:39.823076 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:39.823080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:39.823224 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:39.823317 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:39.825125 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:39.826090 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:39.827776 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:39.828400 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:39.828528 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:39.831488 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:39.831494 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:39.831498 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:39.831544 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:39.834590 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:39.835548 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:39.835554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:39.835556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:39.835744 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:39.836252 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:39.836256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:39.836258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:39.836784 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:39.837022 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:39.837396 (+ 374us) server_negotiation.cc:299] Negotiation successful
0924 03:26:39.837696 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 03:26:39.843529 29664 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:26:39.852731 29664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:26:39.883445 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:39.869360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:36677 (local address 127.0.0.1:33764)
0924 03:26:39.870430 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:39.870499 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:39.871034 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:39.873266 (+ 2232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:39.873277 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:39.873328 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:39.873872 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:39.873889 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:39.877001 (+ 3112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:39.877010 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:39.880074 (+ 3064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:39.880091 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:39.882190 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:39.882196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:39.882459 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:39.882528 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 03:26:39.882687 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 03:26:39.882958 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":512,"threads_started":1}
I20260924 03:26:39.884519 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:39.869639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33764 (local address 127.10.12.131:36677)
0924 03:26:39.872321 (+ 2682us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:39.872334 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:39.872380 (+ 46us) server_negotiation.cc:407] Connection header received
0924 03:26:39.872539 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:39.872554 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:39.872933 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:39.874160 (+ 1227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:39.874598 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:39.876775 (+ 2177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:39.880270 (+ 3495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:39.881962 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:39.882116 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:39.882385 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:39.883827 (+ 1442us) server_negotiation.cc:299] Negotiation successful
0924 03:26:39.884092 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2424,"thread_start_us":1830,"threads_started":1}
I20260924 03:26:39.887859 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:39.870271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:35741 (local address 127.0.0.1:35560)
0924 03:26:39.879208 (+ 8937us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:39.879231 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:39.879394 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:39.880301 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:39.880307 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:39.880339 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:39.880761 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:39.880772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:39.882685 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:39.882691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:39.884636 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:39.884653 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:39.886794 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:39.886802 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:39.887023 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:39.887066 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 03:26:39.887179 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 03:26:39.887362 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8789,"thread_start_us":8581,"threads_started":1}
I20260924 03:26:39.889078 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:39.871532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35560 (local address 127.10.12.129:35741)
0924 03:26:39.878668 (+ 7136us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:39.878676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:39.879375 (+ 699us) server_negotiation.cc:407] Connection header received
0924 03:26:39.879587 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:39.879598 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:39.880044 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:39.880323 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:39.880983 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:39.882548 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:39.884893 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:39.886094 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:39.886276 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:39.886593 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:39.888411 (+ 1818us) server_negotiation.cc:299] Negotiation successful
0924 03:26:39.888702 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6946,"thread_start_us":6771,"threads_started":1}
I20260924 03:26:39.889703 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:39.871254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:44971 (local address 127.0.0.1:49874)
0924 03:26:39.880082 (+ 8828us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:39.880107 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:39.880383 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:39.881710 (+ 1327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:39.881715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:39.881741 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:39.882119 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:39.882129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:39.884083 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:39.884092 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:39.886308 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:39.886326 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:39.888833 (+ 2507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:39.888838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:39.889007 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:39.889039 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 03:26:39.889125 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 03:26:39.889303 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8664,"spinlock_wait_cycles":3328,"thread_start_us":8500,"threads_started":1}
I20260924 03:26:39.889775 30139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:39.872205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49874 (local address 127.10.12.130:44971)
0924 03:26:39.878088 (+ 5883us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:39.878100 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:39.880249 (+ 2149us) server_negotiation.cc:407] Connection header received
0924 03:26:39.880924 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:39.880933 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:39.881216 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:39.881417 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:39.882328 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:39.883834 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:39.886552 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:39.888264 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:39.888410 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:39.888597 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:39.889233 (+ 636us) server_negotiation.cc:299] Negotiation successful
0924 03:26:39.889456 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5683,"thread_start_us":5551,"threads_started":1}
I20260924 03:26:39.890578 30056 tablet_service.cc:1467] Processing CreateTablet for tablet 291d458a21e84bdf8085890250b0977c (DEFAULT_TABLE table=test-table [id=4d131fe5a94b4a9795b667554374595a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:26:39.893487 29788 tablet_service.cc:1467] Processing CreateTablet for tablet 291d458a21e84bdf8085890250b0977c (DEFAULT_TABLE table=test-table [id=4d131fe5a94b4a9795b667554374595a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:26:39.893605 29922 tablet_service.cc:1467] Processing CreateTablet for tablet 291d458a21e84bdf8085890250b0977c (DEFAULT_TABLE table=test-table [id=4d131fe5a94b4a9795b667554374595a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:26:39.895207 29788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 291d458a21e84bdf8085890250b0977c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:39.895220 29922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 291d458a21e84bdf8085890250b0977c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:39.895227 30056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 291d458a21e84bdf8085890250b0977c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:39.909598 30140 tablet_bootstrap.cc:492] T 291d458a21e84bdf8085890250b0977c P 178242d6473e4e13afe67a3dab206f4b: Bootstrap starting.
I20260924 03:26:39.912307 30142 tablet_bootstrap.cc:492] T 291d458a21e84bdf8085890250b0977c P 81e88a74da3e4ebd8be09bf82e605539: Bootstrap starting.
I20260924 03:26:39.914238 30141 tablet_bootstrap.cc:492] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7: Bootstrap starting.
I20260924 03:26:39.916409 30140 tablet_bootstrap.cc:654] T 291d458a21e84bdf8085890250b0977c P 178242d6473e4e13afe67a3dab206f4b: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:39.916409 30142 tablet_bootstrap.cc:654] T 291d458a21e84bdf8085890250b0977c P 81e88a74da3e4ebd8be09bf82e605539: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:39.917732 30141 tablet_bootstrap.cc:654] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:39.918120 30142 log.cc:824] T 291d458a21e84bdf8085890250b0977c P 81e88a74da3e4ebd8be09bf82e605539: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:39.918124 30140 log.cc:824] T 291d458a21e84bdf8085890250b0977c P 178242d6473e4e13afe67a3dab206f4b: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:39.919100 30141 log.cc:824] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:39.922127 30142 tablet_bootstrap.cc:492] T 291d458a21e84bdf8085890250b0977c P 81e88a74da3e4ebd8be09bf82e605539: No bootstrap required, opened a new log
I20260924 03:26:39.922133 30140 tablet_bootstrap.cc:492] T 291d458a21e84bdf8085890250b0977c P 178242d6473e4e13afe67a3dab206f4b: No bootstrap required, opened a new log
I20260924 03:26:39.922446 30140 ts_tablet_manager.cc:1397] T 291d458a21e84bdf8085890250b0977c P 178242d6473e4e13afe67a3dab206f4b: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260924 03:26:39.922470 30142 ts_tablet_manager.cc:1397] T 291d458a21e84bdf8085890250b0977c P 81e88a74da3e4ebd8be09bf82e605539: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260924 03:26:39.925016 30141 tablet_bootstrap.cc:492] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7: No bootstrap required, opened a new log
I20260924 03:26:39.925486 30141 ts_tablet_manager.cc:1397] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s
I20260924 03:26:39.931589 30142 raft_consensus.cc:348] T 291d458a21e84bdf8085890250b0977c P 81e88a74da3e4ebd8be09bf82e605539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e88a74da3e4ebd8be09bf82e605539" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36677 } } peers { permanent_uuid: "f850a85381b8448181438dd7d44d75a7" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44971 } } peers { permanent_uuid: "178242d6473e4e13afe67a3dab206f4b" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35741 } }
I20260924 03:26:39.932128 30142 raft_consensus.cc:374] T 291d458a21e84bdf8085890250b0977c P 81e88a74da3e4ebd8be09bf82e605539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:39.932241 30142 raft_consensus.cc:729] T 291d458a21e84bdf8085890250b0977c P 81e88a74da3e4ebd8be09bf82e605539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81e88a74da3e4ebd8be09bf82e605539, State: Initialized, Role: FOLLOWER
I20260924 03:26:39.932994 30142 consensus_queue.cc:260] T 291d458a21e84bdf8085890250b0977c P 81e88a74da3e4ebd8be09bf82e605539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e88a74da3e4ebd8be09bf82e605539" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36677 } } peers { permanent_uuid: "f850a85381b8448181438dd7d44d75a7" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44971 } } peers { permanent_uuid: "178242d6473e4e13afe67a3dab206f4b" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35741 } }
I20260924 03:26:39.934314 30121 heartbeater.cc:503] Master 127.10.12.190:36925 was elected leader, sending a full tablet report...
I20260924 03:26:39.934083 30140 raft_consensus.cc:348] T 291d458a21e84bdf8085890250b0977c P 178242d6473e4e13afe67a3dab206f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e88a74da3e4ebd8be09bf82e605539" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36677 } } peers { permanent_uuid: "f850a85381b8448181438dd7d44d75a7" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44971 } } peers { permanent_uuid: "178242d6473e4e13afe67a3dab206f4b" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35741 } }
I20260924 03:26:39.934597 30140 raft_consensus.cc:374] T 291d458a21e84bdf8085890250b0977c P 178242d6473e4e13afe67a3dab206f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:39.934688 30140 raft_consensus.cc:729] T 291d458a21e84bdf8085890250b0977c P 178242d6473e4e13afe67a3dab206f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 178242d6473e4e13afe67a3dab206f4b, State: Initialized, Role: FOLLOWER
I20260924 03:26:39.935153 30142 ts_tablet_manager.cc:1428] T 291d458a21e84bdf8085890250b0977c P 81e88a74da3e4ebd8be09bf82e605539: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260924 03:26:39.935534 30140 consensus_queue.cc:260] T 291d458a21e84bdf8085890250b0977c P 178242d6473e4e13afe67a3dab206f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e88a74da3e4ebd8be09bf82e605539" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36677 } } peers { permanent_uuid: "f850a85381b8448181438dd7d44d75a7" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44971 } } peers { permanent_uuid: "178242d6473e4e13afe67a3dab206f4b" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35741 } }
I20260924 03:26:39.937314 30141 raft_consensus.cc:348] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e88a74da3e4ebd8be09bf82e605539" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36677 } } peers { permanent_uuid: "f850a85381b8448181438dd7d44d75a7" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44971 } } peers { permanent_uuid: "178242d6473e4e13afe67a3dab206f4b" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35741 } }
I20260924 03:26:39.937918 30141 raft_consensus.cc:374] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:39.938023 30141 raft_consensus.cc:729] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f850a85381b8448181438dd7d44d75a7, State: Initialized, Role: FOLLOWER
I20260924 03:26:39.938302 30140 ts_tablet_manager.cc:1428] T 291d458a21e84bdf8085890250b0977c P 178242d6473e4e13afe67a3dab206f4b: Time spent starting tablet: real 0.016s user 0.017s sys 0.001s
I20260924 03:26:39.938884 30141 consensus_queue.cc:260] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e88a74da3e4ebd8be09bf82e605539" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36677 } } peers { permanent_uuid: "f850a85381b8448181438dd7d44d75a7" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44971 } } peers { permanent_uuid: "178242d6473e4e13afe67a3dab206f4b" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35741 } }
I20260924 03:26:39.941236 30141 ts_tablet_manager.cc:1428] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260924 03:26:39.941525 29987 heartbeater.cc:503] Master 127.10.12.190:36925 was elected leader, sending a full tablet report...
W20260924 03:26:39.995121 29854 tablet.cc:2369] T 291d458a21e84bdf8085890250b0977c P 178242d6473e4e13afe67a3dab206f4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:26:40.022739 30122 tablet.cc:2369] T 291d458a21e84bdf8085890250b0977c P 81e88a74da3e4ebd8be09bf82e605539: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:26:40.057520 30148 raft_consensus.cc:482] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:26:40.058014 30148 raft_consensus.cc:504] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e88a74da3e4ebd8be09bf82e605539" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36677 } } peers { permanent_uuid: "f850a85381b8448181438dd7d44d75a7" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44971 } } peers { permanent_uuid: "178242d6473e4e13afe67a3dab206f4b" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35741 } }
I20260924 03:26:40.060389 30148 leader_election.cc:288] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81e88a74da3e4ebd8be09bf82e605539 (127.10.12.131:36677), 178242d6473e4e13afe67a3dab206f4b (127.10.12.129:35741)
I20260924 03:26:40.068890 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:40.060970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:37077 (local address 127.10.12.129:35741)
0924 03:26:40.061307 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:40.061312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:40.062197 (+ 885us) server_negotiation.cc:407] Connection header received
0924 03:26:40.062484 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:40.062490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:40.062584 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:40.062704 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:40.063596 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:40.064506 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:40.066790 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:40.067709 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:40.067851 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:40.067940 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:40.068448 (+ 508us) server_negotiation.cc:299] Negotiation successful
0924 03:26:40.068601 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260924 03:26:40.068885 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:40.061025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:52933 (local address 127.10.12.131:36677)
0924 03:26:40.061297 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:40.061305 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:40.062209 (+ 904us) server_negotiation.cc:407] Connection header received
0924 03:26:40.062484 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:40.062490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:40.062584 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:40.062751 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:40.064137 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:40.065064 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:40.066938 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:40.067794 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:40.068043 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:40.068137 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:40.068447 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 03:26:40.068594 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260924 03:26:40.069118 30150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:40.060909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:36677 (local address 127.10.12.130:52933)
0924 03:26:40.062022 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:40.062045 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:40.062257 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:40.062776 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:40.062780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:40.062798 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:40.063359 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:40.063379 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:40.065216 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:40.065220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:40.066758 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:40.066775 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:40.068042 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:40.068047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:40.068272 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:40.068308 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 03:26:40.068389 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 03:26:40.068575 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"mutex_wait_us":92,"thread_start_us":609,"threads_started":1}
I20260924 03:26:40.069167 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:40.061044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:35741 (local address 127.10.12.130:37077)
0924 03:26:40.062022 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:40.062050 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:40.062297 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:40.062744 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:40.062749 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:40.062790 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:40.063400 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:40.063409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:40.064750 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:40.064756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:40.066590 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:40.066606 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:40.068005 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:40.068012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:40.068258 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:40.068297 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 03:26:40.068388 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 03:26:40.068576 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":94,"spinlock_wait_cycles":57216,"thread_start_us":501,"threads_started":1}
I20260924 03:26:40.070497 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "291d458a21e84bdf8085890250b0977c" candidate_uuid: "f850a85381b8448181438dd7d44d75a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "178242d6473e4e13afe67a3dab206f4b" is_pre_election: true
I20260924 03:26:40.070631 30076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "291d458a21e84bdf8085890250b0977c" candidate_uuid: "f850a85381b8448181438dd7d44d75a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81e88a74da3e4ebd8be09bf82e605539" is_pre_election: true
I20260924 03:26:40.071223 30076 raft_consensus.cc:2454] T 291d458a21e84bdf8085890250b0977c P 81e88a74da3e4ebd8be09bf82e605539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f850a85381b8448181438dd7d44d75a7 in term 0.
I20260924 03:26:40.071250 29808 raft_consensus.cc:2454] T 291d458a21e84bdf8085890250b0977c P 178242d6473e4e13afe67a3dab206f4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f850a85381b8448181438dd7d44d75a7 in term 0.
I20260924 03:26:40.072113 29878 leader_election.cc:302] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 178242d6473e4e13afe67a3dab206f4b, f850a85381b8448181438dd7d44d75a7; no voters:
I20260924 03:26:40.072595 30148 raft_consensus.cc:2781] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:26:40.072818 30148 raft_consensus.cc:482] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:26:40.072934 30148 raft_consensus.cc:3037] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:40.075528 30148 raft_consensus.cc:504] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e88a74da3e4ebd8be09bf82e605539" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36677 } } peers { permanent_uuid: "f850a85381b8448181438dd7d44d75a7" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44971 } } peers { permanent_uuid: "178242d6473e4e13afe67a3dab206f4b" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35741 } }
I20260924 03:26:40.076411 30148 leader_election.cc:288] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7 [CANDIDATE]: Term 1 election: Requested vote from peers 81e88a74da3e4ebd8be09bf82e605539 (127.10.12.131:36677), 178242d6473e4e13afe67a3dab206f4b (127.10.12.129:35741)
I20260924 03:26:40.077191 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "291d458a21e84bdf8085890250b0977c" candidate_uuid: "f850a85381b8448181438dd7d44d75a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "178242d6473e4e13afe67a3dab206f4b"
I20260924 03:26:40.077219 30076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "291d458a21e84bdf8085890250b0977c" candidate_uuid: "f850a85381b8448181438dd7d44d75a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81e88a74da3e4ebd8be09bf82e605539"
I20260924 03:26:40.077469 30076 raft_consensus.cc:3037] T 291d458a21e84bdf8085890250b0977c P 81e88a74da3e4ebd8be09bf82e605539 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:40.077471 29808 raft_consensus.cc:3037] T 291d458a21e84bdf8085890250b0977c P 178242d6473e4e13afe67a3dab206f4b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:40.081275 29808 raft_consensus.cc:2454] T 291d458a21e84bdf8085890250b0977c P 178242d6473e4e13afe67a3dab206f4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f850a85381b8448181438dd7d44d75a7 in term 1.
I20260924 03:26:40.081275 30076 raft_consensus.cc:2454] T 291d458a21e84bdf8085890250b0977c P 81e88a74da3e4ebd8be09bf82e605539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f850a85381b8448181438dd7d44d75a7 in term 1.
I20260924 03:26:40.082016 29878 leader_election.cc:302] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 178242d6473e4e13afe67a3dab206f4b, 81e88a74da3e4ebd8be09bf82e605539, f850a85381b8448181438dd7d44d75a7; no voters:
I20260924 03:26:40.082494 30148 raft_consensus.cc:2781] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:26:40.083177 30148 raft_consensus.cc:686] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7 [term 1 LEADER]: Becoming Leader. State: Replica: f850a85381b8448181438dd7d44d75a7, State: Running, Role: LEADER
I20260924 03:26:40.083820 30148 consensus_queue.cc:237] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e88a74da3e4ebd8be09bf82e605539" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36677 } } peers { permanent_uuid: "f850a85381b8448181438dd7d44d75a7" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44971 } } peers { permanent_uuid: "178242d6473e4e13afe67a3dab206f4b" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35741 } }
I20260924 03:26:40.090502 29664 catalog_manager.cc:5495] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to f850a85381b8448181438dd7d44d75a7 (127.10.12.130). New cstate: current_term: 1 leader_uuid: "f850a85381b8448181438dd7d44d75a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e88a74da3e4ebd8be09bf82e605539" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f850a85381b8448181438dd7d44d75a7" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 44971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "178242d6473e4e13afe67a3dab206f4b" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35741 } health_report { overall_health: UNKNOWN } } }
W20260924 03:26:40.120294 29988 tablet.cc:2369] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:26:40.133857 30139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:40.127143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49878 (local address 127.10.12.130:44971)
0924 03:26:40.127343 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:40.127348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:40.127366 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:26:40.127472 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:40.127478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:40.127583 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:40.127727 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:26:40.128220 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:40.129102 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:40.131111 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:40.131711 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:40.131819 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:40.132507 (+ 688us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:26:40.133214 (+ 707us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:26:40.133347 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:40.133430 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 03:26:40.133517 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 03:26:40.145107 30076 raft_consensus.cc:1261] T 291d458a21e84bdf8085890250b0977c P 81e88a74da3e4ebd8be09bf82e605539 [term 1 FOLLOWER]: Refusing update from remote peer f850a85381b8448181438dd7d44d75a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:26:40.145174 29808 raft_consensus.cc:1261] T 291d458a21e84bdf8085890250b0977c P 178242d6473e4e13afe67a3dab206f4b [term 1 FOLLOWER]: Refusing update from remote peer f850a85381b8448181438dd7d44d75a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:26:40.146763 30151 consensus_queue.cc:1035] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "178242d6473e4e13afe67a3dab206f4b" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:26:40.147253 30148 consensus_queue.cc:1035] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81e88a74da3e4ebd8be09bf82e605539" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:26:40.163434 30155 mvcc.cc:205] Tried to move back new op lower bound from 7332742758980988928 to 7332742758748733440. Current Snapshot: MvccSnapshot[applied={T|T < 7332742758980988928}]
I20260924 03:26:40.184885 29662 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:26:40.185359 29662 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:26:40.191310 29662 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 010a6de3e32f487fb5557b0b874aca27: Sending DeleteTablet for 3 replicas of tablet 291d458a21e84bdf8085890250b0977c
I20260924 03:26:40.193157 30056 tablet_service.cc:1514] Processing DeleteTablet for tablet 291d458a21e84bdf8085890250b0977c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:26:40 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:33764
I20260924 03:26:40.193152 29788 tablet_service.cc:1514] Processing DeleteTablet for tablet 291d458a21e84bdf8085890250b0977c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:26:40 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:35560
I20260924 03:26:40.194046 29922 tablet_service.cc:1514] Processing DeleteTablet for tablet 291d458a21e84bdf8085890250b0977c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:26:40 UTC) from {username='kudu', principal='kudu/127.10.12.190@KRBTEST.COM'} at 127.0.0.1:49874
I20260924 03:26:40.199319 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 29725
I20260924 03:26:40.205786 30166 tablet_replica.cc:330] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7: stopping tablet replica
I20260924 03:26:40.205947 30165 tablet_replica.cc:330] T 291d458a21e84bdf8085890250b0977c P 81e88a74da3e4ebd8be09bf82e605539: stopping tablet replica
I20260924 03:26:40.206619 30165 raft_consensus.cc:2229] T 291d458a21e84bdf8085890250b0977c P 81e88a74da3e4ebd8be09bf82e605539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:26:40.206699 30166 raft_consensus.cc:2229] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:26:40.207376 30165 raft_consensus.cc:2258] T 291d458a21e84bdf8085890250b0977c P 81e88a74da3e4ebd8be09bf82e605539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:26:40.207779 30166 raft_consensus.cc:2258] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:26:40.208374 30164 tablet_replica.cc:330] T 291d458a21e84bdf8085890250b0977c P 178242d6473e4e13afe67a3dab206f4b: stopping tablet replica
I20260924 03:26:40.208910 30164 raft_consensus.cc:2229] T 291d458a21e84bdf8085890250b0977c P 178242d6473e4e13afe67a3dab206f4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:26:40.209537 30164 raft_consensus.cc:2258] T 291d458a21e84bdf8085890250b0977c P 178242d6473e4e13afe67a3dab206f4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:26:40.210412 30165 ts_tablet_manager.cc:1905] T 291d458a21e84bdf8085890250b0977c P 81e88a74da3e4ebd8be09bf82e605539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:26:40.212800 30164 ts_tablet_manager.cc:1905] T 291d458a21e84bdf8085890250b0977c P 178242d6473e4e13afe67a3dab206f4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:26:40.213217 30166 ts_tablet_manager.cc:1905] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:26:40.219122 30165 ts_tablet_manager.cc:1918] T 291d458a21e84bdf8085890250b0977c P 81e88a74da3e4ebd8be09bf82e605539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:26:40.219385 30165 log.cc:1196] T 291d458a21e84bdf8085890250b0977c P 81e88a74da3e4ebd8be09bf82e605539: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/291d458a21e84bdf8085890250b0977c
I20260924 03:26:40.220136 30165 ts_tablet_manager.cc:1939] T 291d458a21e84bdf8085890250b0977c P 81e88a74da3e4ebd8be09bf82e605539: Deleting consensus metadata
I20260924 03:26:40.222831 29650 catalog_manager.cc:4841] TS 81e88a74da3e4ebd8be09bf82e605539 (127.10.12.131:36677): tablet 291d458a21e84bdf8085890250b0977c (table test-table [id=4d131fe5a94b4a9795b667554374595a]) successfully deleted
I20260924 03:26:40.222916 30166 ts_tablet_manager.cc:1918] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:26:40.223281 30164 ts_tablet_manager.cc:1918] T 291d458a21e84bdf8085890250b0977c P 178242d6473e4e13afe67a3dab206f4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:26:40.223367 30166 log.cc:1196] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/291d458a21e84bdf8085890250b0977c
I20260924 03:26:40.223491 30164 log.cc:1196] T 291d458a21e84bdf8085890250b0977c P 178242d6473e4e13afe67a3dab206f4b: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790220067268388-10290-0/minicluster-data/ts-0/wal/wals/291d458a21e84bdf8085890250b0977c
I20260924 03:26:40.224145 30164 ts_tablet_manager.cc:1939] T 291d458a21e84bdf8085890250b0977c P 178242d6473e4e13afe67a3dab206f4b: Deleting consensus metadata
I20260924 03:26:40.224279 30166 ts_tablet_manager.cc:1939] T 291d458a21e84bdf8085890250b0977c P f850a85381b8448181438dd7d44d75a7: Deleting consensus metadata
I20260924 03:26:40.226603 29651 catalog_manager.cc:4841] TS 178242d6473e4e13afe67a3dab206f4b (127.10.12.129:35741): tablet 291d458a21e84bdf8085890250b0977c (table test-table [id=4d131fe5a94b4a9795b667554374595a]) successfully deleted
I20260924 03:26:40.226738 29650 catalog_manager.cc:4841] TS f850a85381b8448181438dd7d44d75a7 (127.10.12.130:44971): tablet 291d458a21e84bdf8085890250b0977c (table test-table [id=4d131fe5a94b4a9795b667554374595a]) successfully deleted
Sep 24 03:26:40 dist-test-slave-00qs krb5kdc[29619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.129@KRBTEST.COM
I20260924 03:26:40.237764 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:40.200605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35574 (local address 127.10.12.129:35741)
0924 03:26:40.200864 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:40.200871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:40.204186 (+ 3315us) server_negotiation.cc:407] Connection header received
0924 03:26:40.205403 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:40.205410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:40.205532 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:40.205626 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:40.210109 (+ 4483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:40.211405 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:40.213565 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:40.214281 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:40.214397 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:40.228871 (+ 14474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:40.228892 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:40.228912 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:40.228976 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:40.233364 (+ 4388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:40.234660 (+ 1296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:40.234671 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:40.234674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:40.234909 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:40.235526 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:40.235532 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:40.235534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:40.236171 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:40.236558 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:40.236942 (+ 384us) server_negotiation.cc:299] Negotiation successful
0924 03:26:40.237247 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 03:26:40.238641 29848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:40.341630 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 29725
I20260924 03:26:40.360945 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 29859
Sep 24 03:26:40 dist-test-slave-00qs krb5kdc[29619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.130@KRBTEST.COM
I20260924 03:26:40.381289 30139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:40.361984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49886 (local address 127.10.12.130:44971)
0924 03:26:40.362219 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:40.362226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:40.362264 (+ 38us) server_negotiation.cc:407] Connection header received
0924 03:26:40.362448 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:40.362454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:40.362575 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:40.362716 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:40.364310 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:40.365667 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:40.367074 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:40.367766 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:40.367906 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:40.372993 (+ 5087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:40.373025 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:40.373054 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:40.373123 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:40.377535 (+ 4412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:40.378479 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:40.378496 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:40.378501 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:40.378731 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:40.379127 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:40.379133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:40.379136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:40.379754 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:40.380220 (+ 466us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:40.380516 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 03:26:40.380825 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 03:26:40.382315 29982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:40.515604 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 29859
I20260924 03:26:40.534610 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 29993
Sep 24 03:26:40 dist-test-slave-00qs krb5kdc[29619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.131@KRBTEST.COM
I20260924 03:26:40.553493 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:40.535590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33778 (local address 127.10.12.131:36677)
0924 03:26:40.535933 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:40.535939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:40.536097 (+ 158us) server_negotiation.cc:407] Connection header received
0924 03:26:40.536186 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:40.536191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:40.536274 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:40.536407 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:40.538005 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:40.538976 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:40.541047 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:40.541505 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:40.541621 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:40.546829 (+ 5208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:40.546852 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:40.546873 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:40.546924 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:40.550315 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:40.551256 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:40.551266 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:40.551269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:40.551453 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:40.551875 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:40.551878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:40.551880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:40.552353 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:40.552681 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:40.552939 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 03:26:40.553172 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260924 03:26:40.554433 30116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:40.653633 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 29993
I20260924 03:26:40.668785 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 29632
I20260924 03:26:40.670131 29694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:40.799994 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 29632
2026-09-24T03:26:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (5085 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T03:26:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:26:40Z Disabled control of system clock
I20260924 03:26:40.892671 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:35561
--webserver_interface=127.10.12.190
--webserver_port=0
--builtin_ntp_servers=127.10.12.148:35187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:35561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:26:41.321997 30175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:41.322428 30175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:41.322522 30175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:41.330917 30175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:26:41.331095 30175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:41.331144 30175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:41.331202 30175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:26:41.331249 30175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:26:41.341985 30175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:35561
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:35561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:41.343739 30175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:41.346024 30175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:41.360303 30180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:41.360241 30181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:41.360275 30183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:41.361088 30175 server_base.cc:1034] running on GCE node
I20260924 03:26:41.362303 30175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:41.364336 30175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:41.365593 30175 hybrid_clock.cc:648] HybridClock initialized: now 1790220401365530 us; error 50 us; skew 500 ppm
I20260924 03:26:41.366060 30175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:26:41.376026 30175 webserver.cc:460] Webserver started at http://127.10.12.190:35361/ using document root <none> and password file <none>
I20260924 03:26:41.377113 30175 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:41.377271 30175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:41.377686 30175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:41.381675 30175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "6e368a8b99054c61a2148a9eb16f43e7"
format_stamp: "Formatted at 2026-09-24 03:26:41 on dist-test-slave-00qs"
server_key: "3877291abc469bc7c06e371535087377"
server_key_iv: "e7029494a2f9d75a11d520fd7f3c9fdd"
server_key_version: "encryptionkey@0"
I20260924 03:26:41.382820 30175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "6e368a8b99054c61a2148a9eb16f43e7"
format_stamp: "Formatted at 2026-09-24 03:26:41 on dist-test-slave-00qs"
server_key: "3877291abc469bc7c06e371535087377"
server_key_iv: "e7029494a2f9d75a11d520fd7f3c9fdd"
server_key_version: "encryptionkey@0"
I20260924 03:26:41.390275 30175 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260924 03:26:41.395153 30189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:41.397394 30175 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260924 03:26:41.397650 30175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "6e368a8b99054c61a2148a9eb16f43e7"
format_stamp: "Formatted at 2026-09-24 03:26:41 on dist-test-slave-00qs"
server_key: "3877291abc469bc7c06e371535087377"
server_key_iv: "e7029494a2f9d75a11d520fd7f3c9fdd"
server_key_version: "encryptionkey@0"
I20260924 03:26:41.397936 30175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:41.419749 30175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:41.420943 30175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:41.421259 30175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:41.443161 30175 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:35561
I20260924 03:26:41.443169 30240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:35561 every 8 connection(s)
I20260924 03:26:41.445045 30175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:26:41.448625 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 30175
I20260924 03:26:41.448813 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:26:41.449427 10290 external_mini_cluster.cc:1428] Setting key 125d0330966cb1edea441d3f1f22595d
I20260924 03:26:41.450639 30241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:41.469473 30244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:41.452003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40534 (local address 127.10.12.190:35561)
0924 03:26:41.453488 (+ 1485us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:41.453508 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:41.453579 (+ 71us) server_negotiation.cc:407] Connection header received
0924 03:26:41.455562 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:41.455606 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:41.456632 (+ 1026us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:41.457274 (+ 642us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:41.458033 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:41.460382 (+ 2349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:41.462121 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:41.463256 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:41.463470 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:41.464857 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:41.464868 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:41.464895 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:41.465883 (+ 988us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:41.466018 (+ 135us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:41.466053 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:41.466203 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:41.466477 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 03:26:41.466990 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":471,"threads_started":1}
I20260924 03:26:41.470580 30241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e368a8b99054c61a2148a9eb16f43e7: Bootstrap starting.
I20260924 03:26:41.474646 30241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e368a8b99054c61a2148a9eb16f43e7: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:41.475906 30241 log.cc:824] T 00000000000000000000000000000000 P 6e368a8b99054c61a2148a9eb16f43e7: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:41.480115 30241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e368a8b99054c61a2148a9eb16f43e7: No bootstrap required, opened a new log
I20260924 03:26:41.489398 30241 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e368a8b99054c61a2148a9eb16f43e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e368a8b99054c61a2148a9eb16f43e7" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 35561 } }
I20260924 03:26:41.489938 30241 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e368a8b99054c61a2148a9eb16f43e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:41.490061 30241 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e368a8b99054c61a2148a9eb16f43e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e368a8b99054c61a2148a9eb16f43e7, State: Initialized, Role: FOLLOWER
I20260924 03:26:41.490826 30241 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e368a8b99054c61a2148a9eb16f43e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e368a8b99054c61a2148a9eb16f43e7" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 35561 } }
I20260924 03:26:41.491092 30241 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e368a8b99054c61a2148a9eb16f43e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:26:41.491245 30241 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e368a8b99054c61a2148a9eb16f43e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:26:41.491430 30241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e368a8b99054c61a2148a9eb16f43e7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:41.494382 30241 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e368a8b99054c61a2148a9eb16f43e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e368a8b99054c61a2148a9eb16f43e7" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 35561 } }
I20260924 03:26:41.495095 30241 leader_election.cc:302] T 00000000000000000000000000000000 P 6e368a8b99054c61a2148a9eb16f43e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e368a8b99054c61a2148a9eb16f43e7; no voters:
I20260924 03:26:41.495849 30241 leader_election.cc:288] T 00000000000000000000000000000000 P 6e368a8b99054c61a2148a9eb16f43e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:26:41.496184 30246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e368a8b99054c61a2148a9eb16f43e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:26:41.497121 30246 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e368a8b99054c61a2148a9eb16f43e7 [term 1 LEADER]: Becoming Leader. State: Replica: 6e368a8b99054c61a2148a9eb16f43e7, State: Running, Role: LEADER
I20260924 03:26:41.498003 30246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e368a8b99054c61a2148a9eb16f43e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e368a8b99054c61a2148a9eb16f43e7" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 35561 } }
I20260924 03:26:41.498971 30241 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e368a8b99054c61a2148a9eb16f43e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:26:41.501753 30248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e368a8b99054c61a2148a9eb16f43e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e368a8b99054c61a2148a9eb16f43e7. Latest consensus state: current_term: 1 leader_uuid: "6e368a8b99054c61a2148a9eb16f43e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e368a8b99054c61a2148a9eb16f43e7" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 35561 } } }
I20260924 03:26:41.502362 30248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e368a8b99054c61a2148a9eb16f43e7 [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:41.503057 30247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e368a8b99054c61a2148a9eb16f43e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e368a8b99054c61a2148a9eb16f43e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e368a8b99054c61a2148a9eb16f43e7" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 35561 } } }
I20260924 03:26:41.503440 30247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e368a8b99054c61a2148a9eb16f43e7 [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:41.503484 30253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:26:41.515012 30253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:26:41.526727 30253 catalog_manager.cc:1347] Generated new cluster ID: 60ef869281914c6386823ae3f3f758cb
I20260924 03:26:41.526897 30253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:26:41.544929 30253 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:26:41.547179 30253 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:26:41.562181 30253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e368a8b99054c61a2148a9eb16f43e7: Generated new TSK 0
I20260924 03:26:41.563699 30253 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:26:41.578425 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.129:0
--local_ip_for_outbound_sockets=127.10.12.129
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:35561
--builtin_ntp_servers=127.10.12.148:35187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:26:41.975207 30265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:41.975507 30265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:41.975636 30265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:41.985721 30265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:41.985850 30265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:41.986003 30265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.129
I20260924 03:26:41.995797 30265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.12.129
--webserver_port=0
--tserver_master_addrs=127.10.12.190:35561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.129
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:41.997225 30265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:41.999230 30265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:42.015249 30271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:42.015177 30273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:42.015208 30270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:42.016085 30265 server_base.cc:1034] running on GCE node
I20260924 03:26:42.016776 30265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:42.017839 30265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:42.019086 30265 hybrid_clock.cc:648] HybridClock initialized: now 1790220402019039 us; error 35 us; skew 500 ppm
I20260924 03:26:42.019508 30265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:26:42.022145 30265 webserver.cc:460] Webserver started at http://127.10.12.129:37329/ using document root <none> and password file <none>
I20260924 03:26:42.022923 30265 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:42.023034 30265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:42.023365 30265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:42.026870 30265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-0/data/instance:
uuid: "2b0b3c81ed924723b0d951fc53d7fa5d"
format_stamp: "Formatted at 2026-09-24 03:26:42 on dist-test-slave-00qs"
server_key: "304da1d878b024edc831342fea4dfa23"
server_key_iv: "58b7b63b65e3e1bf2adb631dfa354778"
server_key_version: "encryptionkey@0"
I20260924 03:26:42.027846 30265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance:
uuid: "2b0b3c81ed924723b0d951fc53d7fa5d"
format_stamp: "Formatted at 2026-09-24 03:26:42 on dist-test-slave-00qs"
server_key: "304da1d878b024edc831342fea4dfa23"
server_key_iv: "58b7b63b65e3e1bf2adb631dfa354778"
server_key_version: "encryptionkey@0"
I20260924 03:26:42.033701 30265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 03:26:42.037968 30279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:42.039517 30265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 03:26:42.039696 30265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-0/wal
uuid: "2b0b3c81ed924723b0d951fc53d7fa5d"
format_stamp: "Formatted at 2026-09-24 03:26:42 on dist-test-slave-00qs"
server_key: "304da1d878b024edc831342fea4dfa23"
server_key_iv: "58b7b63b65e3e1bf2adb631dfa354778"
server_key_version: "encryptionkey@0"
I20260924 03:26:42.039923 30265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:42.059480 30265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:42.060485 30265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:42.060849 30265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:42.061934 30265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:26:42.063546 30265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:26:42.063635 30265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:42.063720 30265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:26:42.063773 30265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:42.098417 30265 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.129:35837
I20260924 03:26:42.098515 30391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.129:35837 every 8 connection(s)
I20260924 03:26:42.099872 30265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-0/data/info.pb
I20260924 03:26:42.103111 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 30265
I20260924 03:26:42.103283 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-0/wal/instance
I20260924 03:26:42.103940 10290 external_mini_cluster.cc:1428] Setting key 1a678bf2529a0ec7e21b1e05c067d009
I20260924 03:26:42.107373 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.130:0
--local_ip_for_outbound_sockets=127.10.12.130
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:35561
--builtin_ntp_servers=127.10.12.148:35187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:26:42.116846 30394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:42.103252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:58749 (local address 127.10.12.190:35561)
0924 03:26:42.103898 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:42.103906 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:42.105402 (+ 1496us) server_negotiation.cc:407] Connection header received
0924 03:26:42.107802 (+ 2400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:42.107807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:42.107902 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:42.108049 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:42.110201 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:42.111298 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:42.113501 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:42.113970 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:42.114097 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:42.115888 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:42.115895 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:42.115898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:42.115922 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:42.115999 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:42.116018 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:42.116102 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:42.116437 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 03:26:42.116551 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":261,"threads_started":1}
I20260924 03:26:42.118234 30395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:42.104130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:35561 (local address 127.10.12.129:58749)
0924 03:26:42.105247 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:42.105293 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:42.107437 (+ 2144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:42.108602 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:42.108618 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:42.108791 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:26:42.109806 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:42.109834 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:42.111626 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:42.111632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:42.113269 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:42.113290 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:42.114231 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:42.114239 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:42.114502 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:42.115377 (+ 875us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:26:42.115381 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:26:42.115450 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:26:42.115589 (+ 139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:26:42.116118 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:26:42.116130 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:26:42.116139 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 03:26:42.116372 (+ 233us) client_negotiation.cc:240] Negotiation successful
0924 03:26:42.116835 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":314,"threads_started":1}
I20260924 03:26:42.120152 30392 heartbeater.cc:347] Connected to a master server at 127.10.12.190:35561
I20260924 03:26:42.120493 30392 heartbeater.cc:464] Registering TS with master...
I20260924 03:26:42.121364 30392 heartbeater.cc:511] Master 127.10.12.190:35561 requested a full tablet report, sending...
I20260924 03:26:42.124055 30206 ts_manager.cc:194] Registered new tserver with Master: 2b0b3c81ed924723b0d951fc53d7fa5d (127.10.12.129:35837)
I20260924 03:26:42.126135 30206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.12.129:58749
W20260924 03:26:42.456779 30396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:42.457079 30396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:42.457211 30396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:42.464664 30396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:42.464788 30396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:42.464942 30396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.130
I20260924 03:26:42.474865 30396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.12.130
--webserver_port=0
--tserver_master_addrs=127.10.12.190:35561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.130
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:42.476279 30396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:42.478044 30396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:42.491050 30401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:42.491019 30402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:42.491083 30404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:42.492049 30396 server_base.cc:1034] running on GCE node
I20260924 03:26:42.492708 30396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:42.493736 30396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:42.494975 30396 hybrid_clock.cc:648] HybridClock initialized: now 1790220402494925 us; error 44 us; skew 500 ppm
I20260924 03:26:42.495319 30396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:26:42.497746 30396 webserver.cc:460] Webserver started at http://127.10.12.130:41761/ using document root <none> and password file <none>
I20260924 03:26:42.498476 30396 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:42.498572 30396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:42.498873 30396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:42.501983 30396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-1/data/instance:
uuid: "7a0671c7ae254aaeab1100febab86a91"
format_stamp: "Formatted at 2026-09-24 03:26:42 on dist-test-slave-00qs"
server_key: "2e3bd0d8273d3a8931a70142f1c212bb"
server_key_iv: "894218873ccb665d78f75fb2fd762d3c"
server_key_version: "encryptionkey@0"
I20260924 03:26:42.502822 30396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance:
uuid: "7a0671c7ae254aaeab1100febab86a91"
format_stamp: "Formatted at 2026-09-24 03:26:42 on dist-test-slave-00qs"
server_key: "2e3bd0d8273d3a8931a70142f1c212bb"
server_key_iv: "894218873ccb665d78f75fb2fd762d3c"
server_key_version: "encryptionkey@0"
I20260924 03:26:42.508188 30396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260924 03:26:42.512082 30410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:42.513403 30396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:26:42.513571 30396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-1/wal
uuid: "7a0671c7ae254aaeab1100febab86a91"
format_stamp: "Formatted at 2026-09-24 03:26:42 on dist-test-slave-00qs"
server_key: "2e3bd0d8273d3a8931a70142f1c212bb"
server_key_iv: "894218873ccb665d78f75fb2fd762d3c"
server_key_version: "encryptionkey@0"
I20260924 03:26:42.513778 30396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:42.535218 30396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:42.536144 30396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:42.536453 30396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:42.537483 30396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:26:42.538990 30396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:26:42.539078 30396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:42.539162 30396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:26:42.539214 30396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:42.574115 30396 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.130:34627
I20260924 03:26:42.574208 30522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.130:34627 every 8 connection(s)
I20260924 03:26:42.575753 30396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-1/data/info.pb
I20260924 03:26:42.577584 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 30396
I20260924 03:26:42.577756 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-1/wal/instance
I20260924 03:26:42.578285 10290 external_mini_cluster.cc:1428] Setting key 0411faf20d1710a31b8d2b68dbe83891
I20260924 03:26:42.582058 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.12.131:0
--local_ip_for_outbound_sockets=127.10.12.131
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:35561
--builtin_ntp_servers=127.10.12.148:35187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:26:42.591578 30394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:42.581805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.130:49203 (local address 127.10.12.190:35561)
0924 03:26:42.581991 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:42.581997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:42.582521 (+ 524us) server_negotiation.cc:407] Connection header received
0924 03:26:42.584519 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:42.584525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:42.584618 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:42.584714 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:42.585909 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:42.586668 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:42.588400 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:42.588845 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:42.588991 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:42.590523 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:42.590528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:42.590532 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:42.590553 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:42.590587 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:42.590600 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:42.590676 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:42.591078 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 03:26:42.591237 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 03:26:42.592795 30525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:42.581244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:35561 (local address 127.10.12.130:49203)
0924 03:26:42.582334 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:42.582390 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:42.584217 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:42.584965 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:42.584977 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:42.585078 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:26:42.585719 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:42.585737 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:42.586836 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:42.586840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:42.588238 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:42.588254 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:42.589113 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:42.589119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:42.589360 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:42.590110 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:26:42.590114 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:26:42.590169 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:26:42.590284 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:26:42.590705 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:26:42.590715 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:26:42.590723 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 03:26:42.590911 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 03:26:42.591362 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":638,"threads_started":1}
I20260924 03:26:42.594625 30523 heartbeater.cc:347] Connected to a master server at 127.10.12.190:35561
I20260924 03:26:42.594974 30523 heartbeater.cc:464] Registering TS with master...
I20260924 03:26:42.595810 30523 heartbeater.cc:511] Master 127.10.12.190:35561 requested a full tablet report, sending...
I20260924 03:26:42.597628 30206 ts_manager.cc:194] Registered new tserver with Master: 7a0671c7ae254aaeab1100febab86a91 (127.10.12.130:34627)
I20260924 03:26:42.598456 30206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.12.130:49203
W20260924 03:26:42.948761 30526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:42.949150 30526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:42.949280 30526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:42.957827 30526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:42.957962 30526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:42.958117 30526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.12.131
I20260924 03:26:42.969034 30526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.12.131
--webserver_port=0
--tserver_master_addrs=127.10.12.190:35561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.12.131
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:42.970887 30526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:42.973183 30526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:42.989421 30532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:42.989421 30531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:42.989516 30534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:42.990624 30526 server_base.cc:1034] running on GCE node
I20260924 03:26:42.991460 30526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:42.992782 30526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:42.994026 30526 hybrid_clock.cc:648] HybridClock initialized: now 1790220402993959 us; error 68 us; skew 500 ppm
I20260924 03:26:42.994411 30526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:26:42.997290 30526 webserver.cc:460] Webserver started at http://127.10.12.131:35569/ using document root <none> and password file <none>
I20260924 03:26:42.998173 30526 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:42.998288 30526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:42.998648 30526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:43.003023 30526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-2/data/instance:
uuid: "b89e10b242ad43a495795c5db9ef92ab"
format_stamp: "Formatted at 2026-09-24 03:26:42 on dist-test-slave-00qs"
server_key: "51b4766714fe7760f2e7e317014ae928"
server_key_iv: "8704ac493b50798f5fc84581b730252b"
server_key_version: "encryptionkey@0"
I20260924 03:26:43.004503 30526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance:
uuid: "b89e10b242ad43a495795c5db9ef92ab"
format_stamp: "Formatted at 2026-09-24 03:26:42 on dist-test-slave-00qs"
server_key: "51b4766714fe7760f2e7e317014ae928"
server_key_iv: "8704ac493b50798f5fc84581b730252b"
server_key_version: "encryptionkey@0"
I20260924 03:26:43.013101 30526 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260924 03:26:43.019047 30540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:43.021260 30526 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 03:26:43.021514 30526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-2/wal
uuid: "b89e10b242ad43a495795c5db9ef92ab"
format_stamp: "Formatted at 2026-09-24 03:26:42 on dist-test-slave-00qs"
server_key: "51b4766714fe7760f2e7e317014ae928"
server_key_iv: "8704ac493b50798f5fc84581b730252b"
server_key_version: "encryptionkey@0"
I20260924 03:26:43.021823 30526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:43.044926 30526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:43.046114 30526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:43.046491 30526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:43.047827 30526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:26:43.050064 30526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:26:43.050191 30526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:43.050329 30526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:26:43.050403 30526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:43.088590 30526 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.131:36527
I20260924 03:26:43.088671 30652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.131:36527 every 8 connection(s)
I20260924 03:26:43.090507 30526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-2/data/info.pb
I20260924 03:26:43.092298 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 30526
I20260924 03:26:43.092512 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-2/wal/instance
I20260924 03:26:43.093082 10290 external_mini_cluster.cc:1428] Setting key 7b9e5c4d3ed45d4ad8cdc93d2b60c302
I20260924 03:26:43.108871 30655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:43.094158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.131:46503 (local address 127.10.12.190:35561)
0924 03:26:43.094913 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:43.094920 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:43.096767 (+ 1847us) server_negotiation.cc:407] Connection header received
0924 03:26:43.099438 (+ 2671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:43.099443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:43.099554 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:43.099694 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:43.101466 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.102529 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.105202 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.105831 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.105962 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:43.107708 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:43.107714 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:43.107718 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:43.107752 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:43.107815 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:43.107831 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:43.107930 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:43.108263 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 03:26:43.108403 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":435,"threads_started":1}
I20260924 03:26:43.110419 30656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:43.095114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:35561 (local address 127.10.12.131:46503)
0924 03:26:43.096565 (+ 1451us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:43.096624 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:43.099017 (+ 2393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:43.100039 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:43.100054 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:43.100215 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:26:43.101180 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:43.101211 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.102721 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.102728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:43.104982 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:43.105003 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.106074 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.106080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:43.106334 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:43.107199 (+ 865us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:26:43.107202 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:26:43.107276 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:26:43.107414 (+ 138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:26:43.107983 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:26:43.107994 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:26:43.108003 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 03:26:43.108216 (+ 213us) client_negotiation.cc:240] Negotiation successful
0924 03:26:43.108731 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":335,"threads_started":1}
I20260924 03:26:43.112308 30653 heartbeater.cc:347] Connected to a master server at 127.10.12.190:35561
I20260924 03:26:43.112703 30653 heartbeater.cc:464] Registering TS with master...
I20260924 03:26:43.113713 30653 heartbeater.cc:511] Master 127.10.12.190:35561 requested a full tablet report, sending...
I20260924 03:26:43.115657 30206 ts_manager.cc:194] Registered new tserver with Master: b89e10b242ad43a495795c5db9ef92ab (127.10.12.131:36527)
I20260924 03:26:43.116720 30206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.12.131:46503
I20260924 03:26:43.123126 10290 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:26:43.129977 30392 heartbeater.cc:503] Master 127.10.12.190:35561 was elected leader, sending a full tablet report...
I20260924 03:26:43.141002 30655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:43.128264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40546 (local address 127.10.12.190:35561)
0924 03:26:43.128512 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:43.128517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:43.128942 (+ 425us) server_negotiation.cc:407] Connection header received
0924 03:26:43.129110 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:43.129115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:43.129199 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:43.129312 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:43.129979 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.130865 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.132456 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.132910 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.133033 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:43.140077 (+ 7044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:43.140084 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:43.140088 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:43.140120 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:43.140175 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:43.140193 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:43.140292 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:43.140466 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 03:26:43.140620 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 03:26:43.146994 30206 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:26:43.155452 30206 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:26:43.196508 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:43.171446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:35837 (local address 127.0.0.1:51630)
0924 03:26:43.184057 (+ 12611us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:43.184136 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:43.184509 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:43.187298 (+ 2789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:43.187309 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:43.187354 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:43.188118 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:43.188138 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.190108 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.190114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:43.192855 (+ 2741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:43.192874 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.195216 (+ 2342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.195224 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:43.195468 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:43.195537 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 03:26:43.195718 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 03:26:43.196040 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11951,"mutex_wait_us":1,"thread_start_us":15148,"threads_started":1}
I20260924 03:26:43.197847 30670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:43.172900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60636 (local address 127.10.12.131:36527)
0924 03:26:43.187346 (+ 14446us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:43.187355 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:43.187393 (+ 38us) server_negotiation.cc:407] Connection header received
0924 03:26:43.187514 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:43.187522 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:43.187793 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:43.188063 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:43.189223 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.191080 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.194120 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.195322 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.195479 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:43.195765 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:43.197098 (+ 1333us) server_negotiation.cc:299] Negotiation successful
0924 03:26:43.197403 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14236,"thread_start_us":14050,"threads_started":1}
I20260924 03:26:43.197886 30671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:43.173599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48174 (local address 127.10.12.130:34627)
0924 03:26:43.186763 (+ 13164us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:43.186771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:43.187575 (+ 804us) server_negotiation.cc:407] Connection header received
0924 03:26:43.187698 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:43.187708 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:43.187991 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:43.188187 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:43.189280 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.191106 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.194361 (+ 3255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.195890 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.196075 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:43.196333 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:43.197281 (+ 948us) server_negotiation.cc:299] Negotiation successful
0924 03:26:43.197483 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12987,"thread_start_us":12859,"threads_started":1}
I20260924 03:26:43.197886 30668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:43.171819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51630 (local address 127.10.12.129:35837)
0924 03:26:43.186392 (+ 14573us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:43.186404 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:43.186450 (+ 46us) server_negotiation.cc:407] Connection header received
0924 03:26:43.186592 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:43.186601 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:43.186968 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:43.188363 (+ 1395us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:43.188655 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.189958 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.193182 (+ 3224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.194974 (+ 1792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.195140 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:43.195529 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:43.197280 (+ 1751us) server_negotiation.cc:299] Negotiation successful
0924 03:26:43.197484 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14323,"thread_start_us":14153,"threads_started":1}
I20260924 03:26:43.196688 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:43.171731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:36527 (local address 127.0.0.1:60636)
0924 03:26:43.186324 (+ 14593us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:43.186350 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:43.186537 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:43.188387 (+ 1850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:43.188392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:43.188417 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:43.188971 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:43.188985 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.191272 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.191277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:43.193837 (+ 2560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:43.193855 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.195702 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.195710 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:43.195913 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:43.195978 (+ 65us) client_negotiation.cc:769] Sending connection context
0924 03:26:43.196058 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 03:26:43.196225 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14422,"mutex_wait_us":132,"thread_start_us":15494,"threads_started":1}
I20260924 03:26:43.198618 30667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:43.172231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:34627 (local address 127.0.0.1:48174)
0924 03:26:43.187271 (+ 15040us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:43.187291 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:43.187434 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:43.188483 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:43.188488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:43.188513 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:43.189076 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:43.189087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.191326 (+ 2239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.191331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:43.194084 (+ 2753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:43.194100 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.196131 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.196138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:43.196846 (+ 708us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:43.196878 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 03:26:43.196964 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 03:26:43.197131 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14898,"mutex_wait_us":43,"thread_start_us":12593,"threads_started":1}
I20260924 03:26:43.203110 30458 tablet_service.cc:1467] Processing CreateTablet for tablet b7045e43d4194df08776833fd39b262c (DEFAULT_TABLE table=test-table [id=4348cea206a94946ba436b4798ca3f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:26:43.203117 30327 tablet_service.cc:1467] Processing CreateTablet for tablet b7045e43d4194df08776833fd39b262c (DEFAULT_TABLE table=test-table [id=4348cea206a94946ba436b4798ca3f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:26:43.203111 30588 tablet_service.cc:1467] Processing CreateTablet for tablet b7045e43d4194df08776833fd39b262c (DEFAULT_TABLE table=test-table [id=4348cea206a94946ba436b4798ca3f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:26:43.204797 30458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7045e43d4194df08776833fd39b262c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:43.204850 30588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7045e43d4194df08776833fd39b262c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:43.204850 30327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7045e43d4194df08776833fd39b262c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:43.219980 30672 tablet_bootstrap.cc:492] T b7045e43d4194df08776833fd39b262c P 7a0671c7ae254aaeab1100febab86a91: Bootstrap starting.
I20260924 03:26:43.224710 30672 tablet_bootstrap.cc:654] T b7045e43d4194df08776833fd39b262c P 7a0671c7ae254aaeab1100febab86a91: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:43.226359 30672 log.cc:824] T b7045e43d4194df08776833fd39b262c P 7a0671c7ae254aaeab1100febab86a91: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:43.228112 30673 tablet_bootstrap.cc:492] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d: Bootstrap starting.
I20260924 03:26:43.230450 30672 tablet_bootstrap.cc:492] T b7045e43d4194df08776833fd39b262c P 7a0671c7ae254aaeab1100febab86a91: No bootstrap required, opened a new log
I20260924 03:26:43.230849 30672 ts_tablet_manager.cc:1397] T b7045e43d4194df08776833fd39b262c P 7a0671c7ae254aaeab1100febab86a91: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260924 03:26:43.228132 30674 tablet_bootstrap.cc:492] T b7045e43d4194df08776833fd39b262c P b89e10b242ad43a495795c5db9ef92ab: Bootstrap starting.
I20260924 03:26:43.232529 30673 tablet_bootstrap.cc:654] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:43.234321 30673 log.cc:824] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:43.237432 30674 tablet_bootstrap.cc:654] T b7045e43d4194df08776833fd39b262c P b89e10b242ad43a495795c5db9ef92ab: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:43.238792 30674 log.cc:824] T b7045e43d4194df08776833fd39b262c P b89e10b242ad43a495795c5db9ef92ab: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:43.239560 30673 tablet_bootstrap.cc:492] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d: No bootstrap required, opened a new log
I20260924 03:26:43.239939 30673 ts_tablet_manager.cc:1397] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260924 03:26:43.241293 30672 raft_consensus.cc:348] T b7045e43d4194df08776833fd39b262c P 7a0671c7ae254aaeab1100febab86a91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0b3c81ed924723b0d951fc53d7fa5d" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35837 } } peers { permanent_uuid: "b89e10b242ad43a495795c5db9ef92ab" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36527 } } peers { permanent_uuid: "7a0671c7ae254aaeab1100febab86a91" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 34627 } }
I20260924 03:26:43.241847 30672 raft_consensus.cc:374] T b7045e43d4194df08776833fd39b262c P 7a0671c7ae254aaeab1100febab86a91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:43.241953 30672 raft_consensus.cc:729] T b7045e43d4194df08776833fd39b262c P 7a0671c7ae254aaeab1100febab86a91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0671c7ae254aaeab1100febab86a91, State: Initialized, Role: FOLLOWER
I20260924 03:26:43.242203 30674 tablet_bootstrap.cc:492] T b7045e43d4194df08776833fd39b262c P b89e10b242ad43a495795c5db9ef92ab: No bootstrap required, opened a new log
I20260924 03:26:43.242789 30674 ts_tablet_manager.cc:1397] T b7045e43d4194df08776833fd39b262c P b89e10b242ad43a495795c5db9ef92ab: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20260924 03:26:43.242902 30672 consensus_queue.cc:260] T b7045e43d4194df08776833fd39b262c P 7a0671c7ae254aaeab1100febab86a91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0b3c81ed924723b0d951fc53d7fa5d" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35837 } } peers { permanent_uuid: "b89e10b242ad43a495795c5db9ef92ab" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36527 } } peers { permanent_uuid: "7a0671c7ae254aaeab1100febab86a91" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 34627 } }
I20260924 03:26:43.244638 30523 heartbeater.cc:503] Master 127.10.12.190:35561 was elected leader, sending a full tablet report...
I20260924 03:26:43.245512 30672 ts_tablet_manager.cc:1428] T b7045e43d4194df08776833fd39b262c P 7a0671c7ae254aaeab1100febab86a91: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260924 03:26:43.253831 30673 raft_consensus.cc:348] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0b3c81ed924723b0d951fc53d7fa5d" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35837 } } peers { permanent_uuid: "b89e10b242ad43a495795c5db9ef92ab" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36527 } } peers { permanent_uuid: "7a0671c7ae254aaeab1100febab86a91" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 34627 } }
I20260924 03:26:43.254539 30673 raft_consensus.cc:374] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:43.254681 30673 raft_consensus.cc:729] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0b3c81ed924723b0d951fc53d7fa5d, State: Initialized, Role: FOLLOWER
I20260924 03:26:43.255630 30673 consensus_queue.cc:260] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0b3c81ed924723b0d951fc53d7fa5d" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35837 } } peers { permanent_uuid: "b89e10b242ad43a495795c5db9ef92ab" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36527 } } peers { permanent_uuid: "7a0671c7ae254aaeab1100febab86a91" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 34627 } }
I20260924 03:26:43.256897 30674 raft_consensus.cc:348] T b7045e43d4194df08776833fd39b262c P b89e10b242ad43a495795c5db9ef92ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0b3c81ed924723b0d951fc53d7fa5d" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35837 } } peers { permanent_uuid: "b89e10b242ad43a495795c5db9ef92ab" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36527 } } peers { permanent_uuid: "7a0671c7ae254aaeab1100febab86a91" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 34627 } }
I20260924 03:26:43.257529 30674 raft_consensus.cc:374] T b7045e43d4194df08776833fd39b262c P b89e10b242ad43a495795c5db9ef92ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:43.257663 30674 raft_consensus.cc:729] T b7045e43d4194df08776833fd39b262c P b89e10b242ad43a495795c5db9ef92ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89e10b242ad43a495795c5db9ef92ab, State: Initialized, Role: FOLLOWER
I20260924 03:26:43.258198 30673 ts_tablet_manager.cc:1428] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d: Time spent starting tablet: real 0.018s user 0.011s sys 0.005s
I20260924 03:26:43.258646 30674 consensus_queue.cc:260] T b7045e43d4194df08776833fd39b262c P b89e10b242ad43a495795c5db9ef92ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0b3c81ed924723b0d951fc53d7fa5d" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35837 } } peers { permanent_uuid: "b89e10b242ad43a495795c5db9ef92ab" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36527 } } peers { permanent_uuid: "7a0671c7ae254aaeab1100febab86a91" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 34627 } }
I20260924 03:26:43.260082 30653 heartbeater.cc:503] Master 127.10.12.190:35561 was elected leader, sending a full tablet report...
I20260924 03:26:43.261698 30674 ts_tablet_manager.cc:1428] T b7045e43d4194df08776833fd39b262c P b89e10b242ad43a495795c5db9ef92ab: Time spent starting tablet: real 0.019s user 0.017s sys 0.000s
W20260924 03:26:43.328286 30524 tablet.cc:2369] T b7045e43d4194df08776833fd39b262c P 7a0671c7ae254aaeab1100febab86a91: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:26:43.342914 30654 tablet.cc:2369] T b7045e43d4194df08776833fd39b262c P b89e10b242ad43a495795c5db9ef92ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:26:43.353018 30393 tablet.cc:2369] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:26:43.395191 30679 raft_consensus.cc:482] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:26:43.395705 30679 raft_consensus.cc:504] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0b3c81ed924723b0d951fc53d7fa5d" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35837 } } peers { permanent_uuid: "b89e10b242ad43a495795c5db9ef92ab" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36527 } } peers { permanent_uuid: "7a0671c7ae254aaeab1100febab86a91" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 34627 } }
I20260924 03:26:43.397919 30679 leader_election.cc:288] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b89e10b242ad43a495795c5db9ef92ab (127.10.12.131:36527), 7a0671c7ae254aaeab1100febab86a91 (127.10.12.130:34627)
I20260924 03:26:43.408900 30671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:43.398638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:33203 (local address 127.10.12.130:34627)
0924 03:26:43.398846 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:43.398852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:43.399862 (+ 1010us) server_negotiation.cc:407] Connection header received
0924 03:26:43.400264 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:43.400272 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:43.400378 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:43.400509 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:43.401512 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.402710 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.406164 (+ 3454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.407460 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.407676 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:43.407764 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:43.408375 (+ 611us) server_negotiation.cc:299] Negotiation successful
0924 03:26:43.408560 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 03:26:43.408926 30670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:43.398510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.12.129:59569 (local address 127.10.12.131:36527)
0924 03:26:43.398925 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:43.398934 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:43.399865 (+ 931us) server_negotiation.cc:407] Connection header received
0924 03:26:43.400264 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:43.400272 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:43.400374 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:43.400490 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:26:43.401476 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.402711 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.406129 (+ 3418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.407468 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.407748 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:43.407821 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:43.408376 (+ 555us) server_negotiation.cc:299] Negotiation successful
0924 03:26:43.408563 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260924 03:26:43.409251 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:43.398368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.131:36527 (local address 127.10.12.129:59569)
0924 03:26:43.399684 (+ 1316us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:43.399708 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:43.399943 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:43.400591 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:43.400597 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:43.400635 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:43.401220 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:43.401238 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.402899 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.402906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:43.405796 (+ 2890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:43.405824 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.407756 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.407770 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:43.408049 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:43.408134 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 03:26:43.408335 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 03:26:43.408589 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1128,"mutex_wait_us":212,"thread_start_us":774,"threads_started":1}
I20260924 03:26:43.409394 30682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:43.398403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.130:34627 (local address 127.10.12.129:33203)
0924 03:26:43.399684 (+ 1281us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:43.399708 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:26:43.399938 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:26:43.400591 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:26:43.400733 (+ 142us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:26:43.400880 (+ 147us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:26:43.401309 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:43.401319 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.402899 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.402906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:43.405796 (+ 2890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:26:43.405824 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.407785 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.407792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:26:43.408049 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:43.408134 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 03:26:43.408335 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 03:26:43.408589 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1140,"mutex_wait_us":236,"spinlock_wait_cycles":48896,"thread_start_us":762,"threads_started":1}
I20260924 03:26:43.410827 30608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7045e43d4194df08776833fd39b262c" candidate_uuid: "2b0b3c81ed924723b0d951fc53d7fa5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89e10b242ad43a495795c5db9ef92ab" is_pre_election: true
I20260924 03:26:43.410827 30478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7045e43d4194df08776833fd39b262c" candidate_uuid: "2b0b3c81ed924723b0d951fc53d7fa5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0671c7ae254aaeab1100febab86a91" is_pre_election: true
I20260924 03:26:43.411582 30608 raft_consensus.cc:2454] T b7045e43d4194df08776833fd39b262c P b89e10b242ad43a495795c5db9ef92ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b0b3c81ed924723b0d951fc53d7fa5d in term 0.
I20260924 03:26:43.411583 30478 raft_consensus.cc:2454] T b7045e43d4194df08776833fd39b262c P 7a0671c7ae254aaeab1100febab86a91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b0b3c81ed924723b0d951fc53d7fa5d in term 0.
I20260924 03:26:43.412554 30283 leader_election.cc:302] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0b3c81ed924723b0d951fc53d7fa5d, b89e10b242ad43a495795c5db9ef92ab; no voters:
I20260924 03:26:43.413174 30679 raft_consensus.cc:2781] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:26:43.413352 30679 raft_consensus.cc:482] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:26:43.413456 30679 raft_consensus.cc:3037] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:43.416101 30679 raft_consensus.cc:504] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0b3c81ed924723b0d951fc53d7fa5d" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35837 } } peers { permanent_uuid: "b89e10b242ad43a495795c5db9ef92ab" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36527 } } peers { permanent_uuid: "7a0671c7ae254aaeab1100febab86a91" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 34627 } }
I20260924 03:26:43.416854 30679 leader_election.cc:288] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d [CANDIDATE]: Term 1 election: Requested vote from peers b89e10b242ad43a495795c5db9ef92ab (127.10.12.131:36527), 7a0671c7ae254aaeab1100febab86a91 (127.10.12.130:34627)
I20260924 03:26:43.417663 30478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7045e43d4194df08776833fd39b262c" candidate_uuid: "2b0b3c81ed924723b0d951fc53d7fa5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0671c7ae254aaeab1100febab86a91"
I20260924 03:26:43.417663 30608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7045e43d4194df08776833fd39b262c" candidate_uuid: "2b0b3c81ed924723b0d951fc53d7fa5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89e10b242ad43a495795c5db9ef92ab"
I20260924 03:26:43.417973 30608 raft_consensus.cc:3037] T b7045e43d4194df08776833fd39b262c P b89e10b242ad43a495795c5db9ef92ab [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:43.417968 30478 raft_consensus.cc:3037] T b7045e43d4194df08776833fd39b262c P 7a0671c7ae254aaeab1100febab86a91 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:43.421836 30608 raft_consensus.cc:2454] T b7045e43d4194df08776833fd39b262c P b89e10b242ad43a495795c5db9ef92ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0b3c81ed924723b0d951fc53d7fa5d in term 1.
I20260924 03:26:43.421836 30478 raft_consensus.cc:2454] T b7045e43d4194df08776833fd39b262c P 7a0671c7ae254aaeab1100febab86a91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0b3c81ed924723b0d951fc53d7fa5d in term 1.
I20260924 03:26:43.422664 30283 leader_election.cc:302] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2b0b3c81ed924723b0d951fc53d7fa5d, 7a0671c7ae254aaeab1100febab86a91, b89e10b242ad43a495795c5db9ef92ab; no voters:
I20260924 03:26:43.423136 30679 raft_consensus.cc:2781] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:26:43.423802 30679 raft_consensus.cc:686] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d [term 1 LEADER]: Becoming Leader. State: Replica: 2b0b3c81ed924723b0d951fc53d7fa5d, State: Running, Role: LEADER
I20260924 03:26:43.424500 30679 consensus_queue.cc:237] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0b3c81ed924723b0d951fc53d7fa5d" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35837 } } peers { permanent_uuid: "b89e10b242ad43a495795c5db9ef92ab" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36527 } } peers { permanent_uuid: "7a0671c7ae254aaeab1100febab86a91" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 34627 } }
I20260924 03:26:43.430809 30206 catalog_manager.cc:5495] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b0b3c81ed924723b0d951fc53d7fa5d (127.10.12.129). New cstate: current_term: 1 leader_uuid: "2b0b3c81ed924723b0d951fc53d7fa5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0b3c81ed924723b0d951fc53d7fa5d" member_type: VOTER last_known_addr { host: "127.10.12.129" port: 35837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89e10b242ad43a495795c5db9ef92ab" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0671c7ae254aaeab1100febab86a91" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 34627 } health_report { overall_health: UNKNOWN } } }
I20260924 03:26:43.454324 30668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:43.447304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51640 (local address 127.10.12.129:35837)
0924 03:26:43.447504 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:43.447509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:43.447529 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:26:43.447793 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:43.447797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:43.447897 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:43.448095 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:26:43.448515 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.449319 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.451389 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.451925 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.452118 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:43.452778 (+ 660us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:26:43.453534 (+ 756us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:26:43.453682 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:43.453820 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 03:26:43.453924 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:26:43.465487 30478 raft_consensus.cc:1261] T b7045e43d4194df08776833fd39b262c P 7a0671c7ae254aaeab1100febab86a91 [term 1 FOLLOWER]: Refusing update from remote peer 2b0b3c81ed924723b0d951fc53d7fa5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:26:43.465487 30608 raft_consensus.cc:1261] T b7045e43d4194df08776833fd39b262c P b89e10b242ad43a495795c5db9ef92ab [term 1 FOLLOWER]: Refusing update from remote peer 2b0b3c81ed924723b0d951fc53d7fa5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:26:43.466930 30683 consensus_queue.cc:1035] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89e10b242ad43a495795c5db9ef92ab" member_type: VOTER last_known_addr { host: "127.10.12.131" port: 36527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:26:43.467506 30679 consensus_queue.cc:1035] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0671c7ae254aaeab1100febab86a91" member_type: VOTER last_known_addr { host: "127.10.12.130" port: 34627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:26:43.490036 30687 mvcc.cc:205] Tried to move back new op lower bound from 7332742772580171776 to 7332742772433076224. Current Snapshot: MvccSnapshot[applied={T|T < 7332742772580171776}]
I20260924 03:26:43.501798 30205 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:26:43.502162 30205 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:26:43.506922 30205 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e368a8b99054c61a2148a9eb16f43e7: Sending DeleteTablet for 3 replicas of tablet b7045e43d4194df08776833fd39b262c
I20260924 03:26:43.508575 30588 tablet_service.cc:1514] Processing DeleteTablet for tablet b7045e43d4194df08776833fd39b262c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:26:43 UTC) from {username='slave'} at 127.0.0.1:60636
I20260924 03:26:43.508663 30327 tablet_service.cc:1514] Processing DeleteTablet for tablet b7045e43d4194df08776833fd39b262c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:26:43 UTC) from {username='slave'} at 127.0.0.1:51630
I20260924 03:26:43.510031 30458 tablet_service.cc:1514] Processing DeleteTablet for tablet b7045e43d4194df08776833fd39b262c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:26:43 UTC) from {username='slave'} at 127.0.0.1:48174
I20260924 03:26:43.510829 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 30265
I20260924 03:26:43.523388 30696 tablet_replica.cc:330] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d: stopping tablet replica
I20260924 03:26:43.527992 30696 raft_consensus.cc:2229] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:26:43.528575 30698 tablet_replica.cc:330] T b7045e43d4194df08776833fd39b262c P 7a0671c7ae254aaeab1100febab86a91: stopping tablet replica
I20260924 03:26:43.529120 30698 raft_consensus.cc:2229] T b7045e43d4194df08776833fd39b262c P 7a0671c7ae254aaeab1100febab86a91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:26:43.529834 30698 raft_consensus.cc:2258] T b7045e43d4194df08776833fd39b262c P 7a0671c7ae254aaeab1100febab86a91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:26:43.532312 30696 raft_consensus.cc:2258] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:26:43.533219 30698 ts_tablet_manager.cc:1905] T b7045e43d4194df08776833fd39b262c P 7a0671c7ae254aaeab1100febab86a91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:26:43.535852 30668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:43.512125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51656 (local address 127.10.12.129:35837)
0924 03:26:43.512331 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:43.512339 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:43.520159 (+ 7820us) server_negotiation.cc:407] Connection header received
0924 03:26:43.520406 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:43.520415 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:43.520588 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:43.520709 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:43.523885 (+ 3176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.525393 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.529739 (+ 4346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.530601 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.530733 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:43.533744 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:43.533758 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:43.533780 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:43.534745 (+ 965us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:43.534836 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:43.534869 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:43.534967 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:43.535279 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 03:26:43.535432 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:26:43.536478 30697 tablet_replica.cc:330] T b7045e43d4194df08776833fd39b262c P b89e10b242ad43a495795c5db9ef92ab: stopping tablet replica
I20260924 03:26:43.535849 30696 ts_tablet_manager.cc:1905] T b7045e43d4194df08776833fd39b262c P 2b0b3c81ed924723b0d951fc53d7fa5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:26:43.537298 30387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:43.538160 30697 raft_consensus.cc:2229] T b7045e43d4194df08776833fd39b262c P b89e10b242ad43a495795c5db9ef92ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:26:43.539019 30697 raft_consensus.cc:2258] T b7045e43d4194df08776833fd39b262c P b89e10b242ad43a495795c5db9ef92ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:26:43.542021 30698 ts_tablet_manager.cc:1918] T b7045e43d4194df08776833fd39b262c P 7a0671c7ae254aaeab1100febab86a91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:26:43.542281 30698 log.cc:1196] T b7045e43d4194df08776833fd39b262c P 7a0671c7ae254aaeab1100febab86a91: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-1/wal/wals/b7045e43d4194df08776833fd39b262c
I20260924 03:26:43.542974 30698 ts_tablet_manager.cc:1939] T b7045e43d4194df08776833fd39b262c P 7a0671c7ae254aaeab1100febab86a91: Deleting consensus metadata
I20260924 03:26:43.544384 30697 ts_tablet_manager.cc:1905] T b7045e43d4194df08776833fd39b262c P b89e10b242ad43a495795c5db9ef92ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:26:43.545547 30191 catalog_manager.cc:4841] TS 7a0671c7ae254aaeab1100febab86a91 (127.10.12.130:34627): tablet b7045e43d4194df08776833fd39b262c (table test-table [id=4348cea206a94946ba436b4798ca3f6a]) successfully deleted
I20260924 03:26:43.550721 30697 ts_tablet_manager.cc:1918] T b7045e43d4194df08776833fd39b262c P b89e10b242ad43a495795c5db9ef92ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:26:43.550927 30697 log.cc:1196] T b7045e43d4194df08776833fd39b262c P b89e10b242ad43a495795c5db9ef92ab: Deleting WAL directory at /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790220067268388-10290-0/minicluster-data/ts-2/wal/wals/b7045e43d4194df08776833fd39b262c
I20260924 03:26:43.551388 30697 ts_tablet_manager.cc:1939] T b7045e43d4194df08776833fd39b262c P b89e10b242ad43a495795c5db9ef92ab: Deleting consensus metadata
I20260924 03:26:43.553561 30193 catalog_manager.cc:4841] TS b89e10b242ad43a495795c5db9ef92ab (127.10.12.131:36527): tablet b7045e43d4194df08776833fd39b262c (table test-table [id=4348cea206a94946ba436b4798ca3f6a]) successfully deleted
I20260924 03:26:43.647498 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 30265
I20260924 03:26:43.666442 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 30396
W20260924 03:26:43.666559 30192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.12.129:35837 (error 108)
I20260924 03:26:43.668947 30667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:43.668325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:35837 (local address 127.0.0.1:51664)
0924 03:26:43.668557 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:43.668711 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:35837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260924 03:26:43.669238 30192 catalog_manager.cc:4584] TS 2b0b3c81ed924723b0d951fc53d7fa5d (127.10.12.129:35837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b7045e43d4194df08776833fd39b262c: Network error: Client connection negotiation failed: client connection to 127.10.12.129:35837: connect: Connection refused (error 111)
I20260924 03:26:43.675081 30671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:43.667477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48176 (local address 127.10.12.130:34627)
0924 03:26:43.667700 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:43.667707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:43.667812 (+ 105us) server_negotiation.cc:407] Connection header received
0924 03:26:43.668043 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:43.668050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:43.668212 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:43.668336 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:43.669017 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.670040 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.671475 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.671923 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.672101 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:43.673285 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:43.673294 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:43.673311 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:43.674041 (+ 730us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:43.674103 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:43.674126 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:43.674241 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:43.674387 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 03:26:43.674502 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 03:26:43.676039 30518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:43.723302 30667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:43.722743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:35837 (local address 127.0.0.1:51666)
0924 03:26:43.722992 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:43.723102 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:35837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 03:26:43.724952 30667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:43.724470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:35837 (local address 127.0.0.1:51680)
0924 03:26:43.724657 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:43.724786 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:35837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 03:26:43.776541 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 30396
I20260924 03:26:43.785789 30667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:43.785187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:35837 (local address 127.0.0.1:51684)
0924 03:26:43.785452 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:43.785559 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:35837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260924 03:26:43.786993 30667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:43.786585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:35837 (local address 127.0.0.1:51686)
0924 03:26:43.786750 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:43.786836 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:35837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 03:26:43.793960 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 30526
I20260924 03:26:43.802438 30670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:43.794911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60644 (local address 127.10.12.131:36527)
0924 03:26:43.795155 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:43.795161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:43.795191 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:26:43.795270 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:43.795275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:43.795410 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:43.795536 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:43.796102 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.796980 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.798617 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:43.799185 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:43.799320 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:43.800763 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:43.800775 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:43.800795 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:43.801562 (+ 767us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:43.801626 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:43.801652 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:43.801769 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:43.801921 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 03:26:43.802045 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 03:26:43.803253 30648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:43.868598 30667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:43.867897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:35837 (local address 127.0.0.1:51688)
0924 03:26:43.868184 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:43.868334 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:35837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260924 03:26:43.869874 30667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:43.869439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.129:35837 (local address 127.0.0.1:51692)
0924 03:26:43.869671 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 03:26:43.869748 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.129:35837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260924 03:26:43.898700 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 30526
I20260924 03:26:43.913067 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 30175
I20260924 03:26:43.914279 30236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:44.031064 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 30175
2026-09-24T03:26:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3225 ms)
[----------] 6 tests from EncryptionPolicyTest (23498 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:26:44 dist-test-slave-00qs krb5kdc[30707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:26:44 dist-test-slave-00qs krb5kdc[30707](info): set up 2 sockets
Sep 24 03:26:44 dist-test-slave-00qs krb5kdc[30707](info): commencing operation
krb5kdc: starting...
W20260924 03:26:45.072773 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:26:45 dist-test-slave-00qs krb5kdc[30707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:26:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:26:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:26:45.333582 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35577
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:26:45.675266 30720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:45.675604 30720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:45.675740 30720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:45.683465 30720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:26:45.683637 30720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:45.683718 30720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:45.683801 30720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:26:45.683867 30720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:26:45.694258 30720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35577
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:45.695672 30720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:45.697086 30720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:45.709474 30726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:45.709677 30725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:45.709733 30728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:45.711406 30720 server_base.cc:1034] running on GCE node
I20260924 03:26:45.712391 30720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:45.714233 30720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:45.715580 30720 hybrid_clock.cc:648] HybridClock initialized: now 1790220405715526 us; error 58 us; skew 500 ppm
I20260924 03:26:45.716081 30720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:26:45 dist-test-slave-00qs krb5kdc[30707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:26:45.719383 30720 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:26:45.721170 30720 webserver.cc:460] Webserver started at http://127.0.0.1:42065/ using document root <none> and password file <none>
I20260924 03:26:45.721954 30720 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:45.722088 30720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:45.722404 30720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:45.725601 30720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "0f3bad8a24e94f099f2f95da1fd17bb4"
format_stamp: "Formatted at 2026-09-24 03:26:45 on dist-test-slave-00qs"
server_key: "072bf84766c3c6bbbedfd512382361f9"
server_key_iv: "91f0b25ac05816f56a80aabdb231486f"
server_key_version: "encryptionkey@0"
I20260924 03:26:45.726397 30720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "0f3bad8a24e94f099f2f95da1fd17bb4"
format_stamp: "Formatted at 2026-09-24 03:26:45 on dist-test-slave-00qs"
server_key: "072bf84766c3c6bbbedfd512382361f9"
server_key_iv: "91f0b25ac05816f56a80aabdb231486f"
server_key_version: "encryptionkey@0"
I20260924 03:26:45.731896 30720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 03:26:45.735877 30735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:45.737370 30720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:26:45.737660 30720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "0f3bad8a24e94f099f2f95da1fd17bb4"
format_stamp: "Formatted at 2026-09-24 03:26:45 on dist-test-slave-00qs"
server_key: "072bf84766c3c6bbbedfd512382361f9"
server_key_iv: "91f0b25ac05816f56a80aabdb231486f"
server_key_version: "encryptionkey@0"
I20260924 03:26:45.737891 30720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:45.762032 30720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:45.766587 30720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:45.766868 30720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:45.785148 30720 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35577
I20260924 03:26:45.785197 30786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35577 every 8 connection(s)
I20260924 03:26:45.786571 30720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:26:45.789901 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 30720
I20260924 03:26:45.790967 30787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:45.795193 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:26:45.796082 10290 external_mini_cluster.cc:1428] Setting key 2d01d26d4ce9ec9194f5ff3812094bd3
I20260924 03:26:45.804527 30787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f3bad8a24e94f099f2f95da1fd17bb4: Bootstrap starting.
I20260924 03:26:45.807900 30787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f3bad8a24e94f099f2f95da1fd17bb4: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:45.808980 30787 log.cc:824] T 00000000000000000000000000000000 P 0f3bad8a24e94f099f2f95da1fd17bb4: Log is configured to *not* fsync() on all Append() calls
Sep 24 03:26:45 dist-test-slave-00qs krb5kdc[30707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 03:26:45.812044 30787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f3bad8a24e94f099f2f95da1fd17bb4: No bootstrap required, opened a new log
I20260924 03:26:45.819589 30790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:45.798362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42077 (local address 127.0.0.1:35577)
0924 03:26:45.799134 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:45.799147 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:45.799188 (+ 41us) server_negotiation.cc:407] Connection header received
0924 03:26:45.800959 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:45.800985 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:45.801445 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:45.801820 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:45.803310 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:45.804757 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:45.806036 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:45.806748 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:45.806923 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:45.811471 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:45.811480 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:45.811497 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:45.811537 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:45.814732 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:45.815659 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:45.815667 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:45.815675 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:45.815858 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:45.816357 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:45.816361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:45.816364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:45.816952 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:45.817291 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:45.817853 (+ 562us) server_negotiation.cc:299] Negotiation successful
0924 03:26:45.818188 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":362,"threads_started":1}
I20260924 03:26:45.820255 30787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f3bad8a24e94f099f2f95da1fd17bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3bad8a24e94f099f2f95da1fd17bb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35577 } }
I20260924 03:26:45.820644 30787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f3bad8a24e94f099f2f95da1fd17bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:45.820741 30787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f3bad8a24e94f099f2f95da1fd17bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3bad8a24e94f099f2f95da1fd17bb4, State: Initialized, Role: FOLLOWER
I20260924 03:26:45.821357 30787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f3bad8a24e94f099f2f95da1fd17bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3bad8a24e94f099f2f95da1fd17bb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35577 } }
I20260924 03:26:45.821652 30787 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f3bad8a24e94f099f2f95da1fd17bb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:26:45.821787 30787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f3bad8a24e94f099f2f95da1fd17bb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:26:45.821959 30787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f3bad8a24e94f099f2f95da1fd17bb4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:45.823973 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:26:45.824666 30787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f3bad8a24e94f099f2f95da1fd17bb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3bad8a24e94f099f2f95da1fd17bb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35577 } }
I20260924 03:26:45.825156 30787 leader_election.cc:302] T 00000000000000000000000000000000 P 0f3bad8a24e94f099f2f95da1fd17bb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f3bad8a24e94f099f2f95da1fd17bb4; no voters:
I20260924 03:26:45.825724 30787 leader_election.cc:288] T 00000000000000000000000000000000 P 0f3bad8a24e94f099f2f95da1fd17bb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:26:45.825887 30792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f3bad8a24e94f099f2f95da1fd17bb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:26:45.827723 30787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f3bad8a24e94f099f2f95da1fd17bb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:26:45.835970 30792 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f3bad8a24e94f099f2f95da1fd17bb4 [term 1 LEADER]: Becoming Leader. State: Replica: 0f3bad8a24e94f099f2f95da1fd17bb4, State: Running, Role: LEADER
I20260924 03:26:45.836714 30792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f3bad8a24e94f099f2f95da1fd17bb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3bad8a24e94f099f2f95da1fd17bb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35577 } }
I20260924 03:26:45.839787 30794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f3bad8a24e94f099f2f95da1fd17bb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f3bad8a24e94f099f2f95da1fd17bb4. Latest consensus state: current_term: 1 leader_uuid: "0f3bad8a24e94f099f2f95da1fd17bb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3bad8a24e94f099f2f95da1fd17bb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35577 } } }
I20260924 03:26:45.840204 30794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f3bad8a24e94f099f2f95da1fd17bb4 [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:45.840667 30793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f3bad8a24e94f099f2f95da1fd17bb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f3bad8a24e94f099f2f95da1fd17bb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3bad8a24e94f099f2f95da1fd17bb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35577 } } }
I20260924 03:26:45.840970 30793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f3bad8a24e94f099f2f95da1fd17bb4 [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:45.841530 30811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:26:45.847932 30811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:26:45.855151 30790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:45.837940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47657 (local address 127.0.0.1:35577)
0924 03:26:45.838157 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:45.838165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:45.838654 (+ 489us) server_negotiation.cc:407] Connection header received
0924 03:26:45.839044 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:45.839050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:45.839140 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:45.839254 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:45.841118 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:45.842092 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:45.843665 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:45.844379 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:45.844512 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:45.847633 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:45.847641 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:45.847649 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:45.847693 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:45.851688 (+ 3995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:45.852733 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:45.852740 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:45.852743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:45.852919 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:45.853447 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:45.853453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:45.853457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:45.853908 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:45.854198 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:45.854619 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 03:26:45.854806 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:26:45.858392 30811 catalog_manager.cc:1347] Generated new cluster ID: a924dc96c98f487e8111a05e1a003bf3
I20260924 03:26:45.858510 30811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:26:45.896720 30811 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:26:45.898149 30811 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:26:45.909646 30811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f3bad8a24e94f099f2f95da1fd17bb4: Generated new TSK 0
I20260924 03:26:45.910590 30811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:26:45.918743 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 30720
I20260924 03:26:45.920136 30782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:46.021008 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 30720
W20260924 03:26:46.036386 30705 connection.cc:535] client connection to 127.0.0.1:35577 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35577): Connection reset by peer (error 104)
2026-09-24T03:26:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1985 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:26:46 dist-test-slave-00qs krb5kdc[30819](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:26:46 dist-test-slave-00qs krb5kdc[30819](info): set up 2 sockets
Sep 24 03:26:46 dist-test-slave-00qs krb5kdc[30819](info): commencing operation
krb5kdc: starting...
W20260924 03:26:47.056440 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:26:47 dist-test-slave-00qs krb5kdc[30819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:26:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:26:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:26:47.316505 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33151
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:26:47.655848 30832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:47.656189 30832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:47.656332 30832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:47.663851 30832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:26:47.664023 30832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:47.664109 30832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:47.664198 30832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:26:47.664261 30832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:26:47.674970 30832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33151
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:47.676436 30832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:47.677783 30832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:47.689858 30837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:47.689832 30838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:47.689893 30840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:47.691078 30832 server_base.cc:1034] running on GCE node
I20260924 03:26:47.691852 30832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:47.693487 30832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:47.694718 30832 hybrid_clock.cc:648] HybridClock initialized: now 1790220407694678 us; error 39 us; skew 500 ppm
I20260924 03:26:47.695104 30832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:26:47 dist-test-slave-00qs krb5kdc[30819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:26:47.703717 30832 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:26:47.705379 30832 webserver.cc:460] Webserver started at http://127.0.0.1:39703/ using document root <none> and password file <none>
I20260924 03:26:47.706075 30832 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:47.706182 30832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:47.706441 30832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:47.709513 30832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "c54142ec94564eccbbf5ff55890f34fb"
format_stamp: "Formatted at 2026-09-24 03:26:47 on dist-test-slave-00qs"
server_key: "0312d1e9bf6ba9b1533a2e3a532d2cea"
server_key_iv: "74016b241feeef9e211cd93395528d50"
server_key_version: "encryptionkey@0"
I20260924 03:26:47.710362 30832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "c54142ec94564eccbbf5ff55890f34fb"
format_stamp: "Formatted at 2026-09-24 03:26:47 on dist-test-slave-00qs"
server_key: "0312d1e9bf6ba9b1533a2e3a532d2cea"
server_key_iv: "74016b241feeef9e211cd93395528d50"
server_key_version: "encryptionkey@0"
I20260924 03:26:47.716014 30832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 03:26:47.720119 30847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:47.721674 30832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 03:26:47.721848 30832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "c54142ec94564eccbbf5ff55890f34fb"
format_stamp: "Formatted at 2026-09-24 03:26:47 on dist-test-slave-00qs"
server_key: "0312d1e9bf6ba9b1533a2e3a532d2cea"
server_key_iv: "74016b241feeef9e211cd93395528d50"
server_key_version: "encryptionkey@0"
I20260924 03:26:47.722050 30832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:47.751600 30832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:47.756309 30832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:47.756558 30832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:47.775426 30832 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33151
I20260924 03:26:47.775446 30898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33151 every 8 connection(s)
I20260924 03:26:47.776878 30832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:26:47.780758 30899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:47.781783 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 30832
I20260924 03:26:47.782094 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:26:47.782552 10290 external_mini_cluster.cc:1428] Setting key 2938fbc39541839b79100410790706c0
I20260924 03:26:47.795802 30899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c54142ec94564eccbbf5ff55890f34fb: Bootstrap starting.
I20260924 03:26:47.797952 30902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:47.784479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58527 (local address 127.0.0.1:33151)
0924 03:26:47.785443 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:47.785464 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:47.785518 (+ 54us) server_negotiation.cc:407] Connection header received
0924 03:26:47.787418 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:47.787459 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:47.788102 (+ 643us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:47.788598 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:47.789129 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:47.791026 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:47.792557 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:47.793737 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:47.793913 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:47.795092 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:47.795103 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:47.795124 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:47.795847 (+ 723us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:47.795986 (+ 139us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:47.796022 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:47.796182 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:47.796428 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 03:26:47.796612 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":377,"threads_started":1}
I20260924 03:26:47.799520 30899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c54142ec94564eccbbf5ff55890f34fb: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:47.800654 30899 log.cc:824] T 00000000000000000000000000000000 P c54142ec94564eccbbf5ff55890f34fb: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:47.802148 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:26:47.803726 30899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c54142ec94564eccbbf5ff55890f34fb: No bootstrap required, opened a new log
I20260924 03:26:47.812219 30899 raft_consensus.cc:348] T 00000000000000000000000000000000 P c54142ec94564eccbbf5ff55890f34fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54142ec94564eccbbf5ff55890f34fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33151 } }
I20260924 03:26:47.812717 30902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:47.806524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39873 (local address 127.0.0.1:33151)
0924 03:26:47.806723 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:47.806730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:47.807022 (+ 292us) server_negotiation.cc:407] Connection header received
0924 03:26:47.807213 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:47.807217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:47.807287 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:47.807379 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:47.807910 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:47.808752 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:47.810081 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:47.810581 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:47.810696 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:47.811928 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:47.811933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:47.811937 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:47.811990 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:47.812055 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:47.812074 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:47.812174 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:47.812371 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 03:26:47.812426 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 03:26:47.812744 30899 raft_consensus.cc:374] T 00000000000000000000000000000000 P c54142ec94564eccbbf5ff55890f34fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:47.812853 30899 raft_consensus.cc:729] T 00000000000000000000000000000000 P c54142ec94564eccbbf5ff55890f34fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c54142ec94564eccbbf5ff55890f34fb, State: Initialized, Role: FOLLOWER
I20260924 03:26:47.813453 30899 consensus_queue.cc:260] T 00000000000000000000000000000000 P c54142ec94564eccbbf5ff55890f34fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54142ec94564eccbbf5ff55890f34fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33151 } }
I20260924 03:26:47.813699 30899 raft_consensus.cc:388] T 00000000000000000000000000000000 P c54142ec94564eccbbf5ff55890f34fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:26:47.813836 30899 raft_consensus.cc:482] T 00000000000000000000000000000000 P c54142ec94564eccbbf5ff55890f34fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:26:47.814003 30899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c54142ec94564eccbbf5ff55890f34fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:47.816361 30899 raft_consensus.cc:504] T 00000000000000000000000000000000 P c54142ec94564eccbbf5ff55890f34fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54142ec94564eccbbf5ff55890f34fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33151 } }
I20260924 03:26:47.816926 30899 leader_election.cc:302] T 00000000000000000000000000000000 P c54142ec94564eccbbf5ff55890f34fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c54142ec94564eccbbf5ff55890f34fb; no voters:
I20260924 03:26:47.817605 30899 leader_election.cc:288] T 00000000000000000000000000000000 P c54142ec94564eccbbf5ff55890f34fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:26:47.817687 30909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c54142ec94564eccbbf5ff55890f34fb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:26:47.818364 30909 raft_consensus.cc:686] T 00000000000000000000000000000000 P c54142ec94564eccbbf5ff55890f34fb [term 1 LEADER]: Becoming Leader. State: Replica: c54142ec94564eccbbf5ff55890f34fb, State: Running, Role: LEADER
I20260924 03:26:47.818948 30909 consensus_queue.cc:237] T 00000000000000000000000000000000 P c54142ec94564eccbbf5ff55890f34fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54142ec94564eccbbf5ff55890f34fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33151 } }
I20260924 03:26:47.819702 30899 sys_catalog.cc:564] T 00000000000000000000000000000000 P c54142ec94564eccbbf5ff55890f34fb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:26:47.821729 30910 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54142ec94564eccbbf5ff55890f34fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c54142ec94564eccbbf5ff55890f34fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54142ec94564eccbbf5ff55890f34fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33151 } } }
I20260924 03:26:47.822124 30910 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54142ec94564eccbbf5ff55890f34fb [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:47.821748 30911 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54142ec94564eccbbf5ff55890f34fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c54142ec94564eccbbf5ff55890f34fb. Latest consensus state: current_term: 1 leader_uuid: "c54142ec94564eccbbf5ff55890f34fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c54142ec94564eccbbf5ff55890f34fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33151 } } }
I20260924 03:26:47.823922 30911 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54142ec94564eccbbf5ff55890f34fb [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:47.826586 30914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:26:47.833261 30914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:26:47.842471 30914 catalog_manager.cc:1347] Generated new cluster ID: 7de87e16e7f143b5b706bf2d5e24f14c
I20260924 03:26:47.842598 30914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:26:47.861912 30914 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:26:47.863837 30914 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:26:47.874899 30914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c54142ec94564eccbbf5ff55890f34fb: Generated new TSK 0
I20260924 03:26:47.876298 30914 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:26:47.886178 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 30832
W20260924 03:26:47.887380 30894 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58527
W20260924 03:26:47.888307 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 30832: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:26:47.888540 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 30832
W20260924 03:26:47.902346 30817 connection.cc:535] client connection to 127.0.0.1:33151 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33151): Connection reset by peer (error 104)
2026-09-24T03:26:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1866 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:26:48 dist-test-slave-00qs krb5kdc[30930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:26:48 dist-test-slave-00qs krb5kdc[30930](info): set up 2 sockets
Sep 24 03:26:48 dist-test-slave-00qs krb5kdc[30930](info): commencing operation
krb5kdc: starting...
W20260924 03:26:48.964536 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.004s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:26:49 dist-test-slave-00qs krb5kdc[30930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:26:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:26:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:26:49.228518 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44343
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:26:49.567476 30943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:49.567786 30943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:49.567909 30943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:49.575471 30943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:26:49.575598 30943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:49.575646 30943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:49.575727 30943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:26:49.575780 30943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:26:49.586040 30943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44343
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:49.587378 30943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:49.588752 30943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:49.601102 30949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:49.601187 30948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:49.601171 30951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:49.601907 30943 server_base.cc:1034] running on GCE node
I20260924 03:26:49.602784 30943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:49.604446 30943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:49.605686 30943 hybrid_clock.cc:648] HybridClock initialized: now 1790220409605634 us; error 50 us; skew 500 ppm
I20260924 03:26:49.606105 30943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:26:49 dist-test-slave-00qs krb5kdc[30930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:26:49.609421 30943 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:26:49.611011 30943 webserver.cc:460] Webserver started at http://127.0.0.1:35315/ using document root <none> and password file <none>
I20260924 03:26:49.611707 30943 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:49.611833 30943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:49.612190 30943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:49.615317 30943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "d97b132508e947e2964208700035ec76"
format_stamp: "Formatted at 2026-09-24 03:26:49 on dist-test-slave-00qs"
server_key: "cf7ca2b178643d72666c847b0df5fb12"
server_key_iv: "68676559afaa221634a3636298ebe8fa"
server_key_version: "encryptionkey@0"
I20260924 03:26:49.616189 30943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "d97b132508e947e2964208700035ec76"
format_stamp: "Formatted at 2026-09-24 03:26:49 on dist-test-slave-00qs"
server_key: "cf7ca2b178643d72666c847b0df5fb12"
server_key_iv: "68676559afaa221634a3636298ebe8fa"
server_key_version: "encryptionkey@0"
I20260924 03:26:49.621639 30943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 03:26:49.625263 30958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:49.626658 30943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:26:49.626847 30943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "d97b132508e947e2964208700035ec76"
format_stamp: "Formatted at 2026-09-24 03:26:49 on dist-test-slave-00qs"
server_key: "cf7ca2b178643d72666c847b0df5fb12"
server_key_iv: "68676559afaa221634a3636298ebe8fa"
server_key_version: "encryptionkey@0"
I20260924 03:26:49.627076 30943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:49.632913 30943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:49.637249 30943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:49.637547 30943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:49.654845 30943 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44343
I20260924 03:26:49.654888 31009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44343 every 8 connection(s)
I20260924 03:26:49.656322 30943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:26:49.660982 31010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:49.664105 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 30943
I20260924 03:26:49.664251 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:26:49.664666 10290 external_mini_cluster.cc:1428] Setting key e556889b524e17584c46ae5127dfd138
I20260924 03:26:49.672473 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:49.666466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41067 (local address 127.0.0.1:44343)
0924 03:26:49.667258 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:49.667269 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:49.667310 (+ 41us) server_negotiation.cc:407] Connection header received
0924 03:26:49.669017 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:49.669041 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:49.669415 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:49.669765 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:49.670850 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:49.670857 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:49.670873 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:49.670887 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:49.670946 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:49.670966 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:49.671135 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:49.671233 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 03:26:49.671405 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":341,"threads_started":1}
I20260924 03:26:49.673062 31010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d97b132508e947e2964208700035ec76: Bootstrap starting.
I20260924 03:26:49.676151 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:26:49.676424 31010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d97b132508e947e2964208700035ec76: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:49.677409 31010 log.cc:824] T 00000000000000000000000000000000 P d97b132508e947e2964208700035ec76: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:49.680361 31010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d97b132508e947e2964208700035ec76: No bootstrap required, opened a new log
I20260924 03:26:49.683377 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:49.680387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59129 (local address 127.0.0.1:44343)
0924 03:26:49.680580 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:49.680587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:49.681088 (+ 501us) server_negotiation.cc:407] Connection header received
0924 03:26:49.681221 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:49.681228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:49.681460 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:49.681559 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:49.682528 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:49.682534 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:49.682539 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:49.682562 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:49.682613 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:49.682633 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:49.682745 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:49.682944 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 03:26:49.683038 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 03:26:49.689163 31010 raft_consensus.cc:348] T 00000000000000000000000000000000 P d97b132508e947e2964208700035ec76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97b132508e947e2964208700035ec76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44343 } }
I20260924 03:26:49.689568 31010 raft_consensus.cc:374] T 00000000000000000000000000000000 P d97b132508e947e2964208700035ec76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:49.689642 31010 raft_consensus.cc:729] T 00000000000000000000000000000000 P d97b132508e947e2964208700035ec76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97b132508e947e2964208700035ec76, State: Initialized, Role: FOLLOWER
I20260924 03:26:49.690161 31010 consensus_queue.cc:260] T 00000000000000000000000000000000 P d97b132508e947e2964208700035ec76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97b132508e947e2964208700035ec76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44343 } }
I20260924 03:26:49.690402 31010 raft_consensus.cc:388] T 00000000000000000000000000000000 P d97b132508e947e2964208700035ec76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:26:49.690503 31010 raft_consensus.cc:482] T 00000000000000000000000000000000 P d97b132508e947e2964208700035ec76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:26:49.690626 31010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d97b132508e947e2964208700035ec76 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:49.693167 31010 raft_consensus.cc:504] T 00000000000000000000000000000000 P d97b132508e947e2964208700035ec76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97b132508e947e2964208700035ec76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44343 } }
I20260924 03:26:49.693674 31010 leader_election.cc:302] T 00000000000000000000000000000000 P d97b132508e947e2964208700035ec76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d97b132508e947e2964208700035ec76; no voters:
I20260924 03:26:49.694389 31010 leader_election.cc:288] T 00000000000000000000000000000000 P d97b132508e947e2964208700035ec76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:26:49.694675 31020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d97b132508e947e2964208700035ec76 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:26:49.695526 31020 raft_consensus.cc:686] T 00000000000000000000000000000000 P d97b132508e947e2964208700035ec76 [term 1 LEADER]: Becoming Leader. State: Replica: d97b132508e947e2964208700035ec76, State: Running, Role: LEADER
I20260924 03:26:49.696254 31020 consensus_queue.cc:237] T 00000000000000000000000000000000 P d97b132508e947e2964208700035ec76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97b132508e947e2964208700035ec76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44343 } }
I20260924 03:26:49.696662 31010 sys_catalog.cc:564] T 00000000000000000000000000000000 P d97b132508e947e2964208700035ec76 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:26:49.699218 31022 sys_catalog.cc:455] T 00000000000000000000000000000000 P d97b132508e947e2964208700035ec76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d97b132508e947e2964208700035ec76. Latest consensus state: current_term: 1 leader_uuid: "d97b132508e947e2964208700035ec76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97b132508e947e2964208700035ec76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44343 } } }
I20260924 03:26:49.699273 31021 sys_catalog.cc:455] T 00000000000000000000000000000000 P d97b132508e947e2964208700035ec76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d97b132508e947e2964208700035ec76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97b132508e947e2964208700035ec76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44343 } } }
I20260924 03:26:49.699862 31021 sys_catalog.cc:458] T 00000000000000000000000000000000 P d97b132508e947e2964208700035ec76 [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:49.699872 31022 sys_catalog.cc:458] T 00000000000000000000000000000000 P d97b132508e947e2964208700035ec76 [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:49.705109 31028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:26:49.709911 31028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:26:49.718487 31028 catalog_manager.cc:1347] Generated new cluster ID: 29887bd6492142e8b57a6fa2d2f6c24b
I20260924 03:26:49.718611 31028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:26:49.755257 31028 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:26:49.755738 31028 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:26:49.765882 31028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d97b132508e947e2964208700035ec76: Generated new TSK 0
I20260924 03:26:49.766799 31028 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:26:49.775578 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 30943
W20260924 03:26:49.776811 31005 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41067
W20260924 03:26:49.777489 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 30943: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:26:49.777637 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 30943
2026-09-24T03:26:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1888 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:26:49 dist-test-slave-00qs krb5kdc[31041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:26:49 dist-test-slave-00qs krb5kdc[31041](info): set up 2 sockets
Sep 24 03:26:49 dist-test-slave-00qs krb5kdc[31041](info): commencing operation
krb5kdc: starting...
W20260924 03:26:50.812466 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:26:50 dist-test-slave-00qs krb5kdc[31041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:26:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:26:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:26:51.072141 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43691
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:26:51.411329 31054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:51.412151 31054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:51.412417 31054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:51.420694 31054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:26:51.420838 31054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:51.420893 31054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:51.420981 31054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:26:51.421036 31054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:26:51.431581 31054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:51.432981 31054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:51.434345 31054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:51.447649 31059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:51.447775 31062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:51.447717 31060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:51.448549 31054 server_base.cc:1034] running on GCE node
I20260924 03:26:51.449405 31054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:51.450963 31054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:51.452180 31054 hybrid_clock.cc:648] HybridClock initialized: now 1790220411452135 us; error 43 us; skew 500 ppm
I20260924 03:26:51.452569 31054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:26:51 dist-test-slave-00qs krb5kdc[31041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:26:51.456040 31054 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:26:51.457679 31054 webserver.cc:460] Webserver started at http://127.0.0.1:43123/ using document root <none> and password file <none>
I20260924 03:26:51.458374 31054 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:51.458479 31054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:51.458734 31054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:51.461884 31054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "370f854c39794cca8b4114af504f5fb6"
format_stamp: "Formatted at 2026-09-24 03:26:51 on dist-test-slave-00qs"
server_key: "2ab8a96909f6f5f919bb18e2ecd783e7"
server_key_iv: "d0e97d3266afb67e2c00a89c711b2780"
server_key_version: "encryptionkey@0"
I20260924 03:26:51.462703 31054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "370f854c39794cca8b4114af504f5fb6"
format_stamp: "Formatted at 2026-09-24 03:26:51 on dist-test-slave-00qs"
server_key: "2ab8a96909f6f5f919bb18e2ecd783e7"
server_key_iv: "d0e97d3266afb67e2c00a89c711b2780"
server_key_version: "encryptionkey@0"
I20260924 03:26:51.468494 31054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:26:51.472561 31069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:51.474066 31054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:26:51.474241 31054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "370f854c39794cca8b4114af504f5fb6"
format_stamp: "Formatted at 2026-09-24 03:26:51 on dist-test-slave-00qs"
server_key: "2ab8a96909f6f5f919bb18e2ecd783e7"
server_key_iv: "d0e97d3266afb67e2c00a89c711b2780"
server_key_version: "encryptionkey@0"
I20260924 03:26:51.474433 31054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:51.504158 31054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:51.508498 31054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:51.508746 31054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:51.526141 31054 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43691
I20260924 03:26:51.526163 31120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43691 every 8 connection(s)
I20260924 03:26:51.527582 31054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:26:51.528726 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 31054
I20260924 03:26:51.528936 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:26:51.529349 10290 external_mini_cluster.cc:1428] Setting key 0092834323dcdfd3339132c8c6fda9cd
I20260924 03:26:51.532193 31121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 03:26:51 dist-test-slave-00qs krb5kdc[31041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 03:26:51.545569 31121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 370f854c39794cca8b4114af504f5fb6: Bootstrap starting.
I20260924 03:26:51.549419 31121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 370f854c39794cca8b4114af504f5fb6: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:51.550678 31121 log.cc:824] T 00000000000000000000000000000000 P 370f854c39794cca8b4114af504f5fb6: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:51.553740 31121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 370f854c39794cca8b4114af504f5fb6: No bootstrap required, opened a new log
I20260924 03:26:51.555403 31124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:51.531254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45033 (local address 127.0.0.1:43691)
0924 03:26:51.532360 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:51.532376 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:51.532430 (+ 54us) server_negotiation.cc:407] Connection header received
0924 03:26:51.534739 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:51.534771 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:51.535334 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:51.535825 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:51.537412 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:51.539491 (+ 2079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:51.540862 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:51.541893 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:51.542103 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:51.546847 (+ 4744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:51.546861 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:51.546885 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:51.546973 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:51.551079 (+ 4106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:51.552023 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:51.552031 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:51.552038 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:51.552231 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:51.552633 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:51.552637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:51.552639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:51.553182 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:51.553489 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:51.553918 (+ 429us) server_negotiation.cc:299] Negotiation successful
0924 03:26:51.554378 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":841,"thread_start_us":636,"threads_started":1}
I20260924 03:26:51.559700 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:26:51.562384 31121 raft_consensus.cc:348] T 00000000000000000000000000000000 P 370f854c39794cca8b4114af504f5fb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370f854c39794cca8b4114af504f5fb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43691 } }
I20260924 03:26:51.562793 31121 raft_consensus.cc:374] T 00000000000000000000000000000000 P 370f854c39794cca8b4114af504f5fb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:51.562928 31121 raft_consensus.cc:729] T 00000000000000000000000000000000 P 370f854c39794cca8b4114af504f5fb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 370f854c39794cca8b4114af504f5fb6, State: Initialized, Role: FOLLOWER
I20260924 03:26:51.563545 31121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 370f854c39794cca8b4114af504f5fb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370f854c39794cca8b4114af504f5fb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43691 } }
I20260924 03:26:51.563787 31121 raft_consensus.cc:388] T 00000000000000000000000000000000 P 370f854c39794cca8b4114af504f5fb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:26:51.563941 31121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 370f854c39794cca8b4114af504f5fb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:26:51.564188 31121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 370f854c39794cca8b4114af504f5fb6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:51.566696 31121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 370f854c39794cca8b4114af504f5fb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370f854c39794cca8b4114af504f5fb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43691 } }
I20260924 03:26:51.567329 31121 leader_election.cc:302] T 00000000000000000000000000000000 P 370f854c39794cca8b4114af504f5fb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 370f854c39794cca8b4114af504f5fb6; no voters:
I20260924 03:26:51.568085 31121 leader_election.cc:288] T 00000000000000000000000000000000 P 370f854c39794cca8b4114af504f5fb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:26:51.568207 31131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 370f854c39794cca8b4114af504f5fb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:26:51.571478 31121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 370f854c39794cca8b4114af504f5fb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:26:51.572253 31131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 370f854c39794cca8b4114af504f5fb6 [term 1 LEADER]: Becoming Leader. State: Replica: 370f854c39794cca8b4114af504f5fb6, State: Running, Role: LEADER
I20260924 03:26:51.572995 31131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 370f854c39794cca8b4114af504f5fb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370f854c39794cca8b4114af504f5fb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43691 } }
I20260924 03:26:51.577345 31132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 370f854c39794cca8b4114af504f5fb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "370f854c39794cca8b4114af504f5fb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370f854c39794cca8b4114af504f5fb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43691 } } }
I20260924 03:26:51.577318 31133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 370f854c39794cca8b4114af504f5fb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 370f854c39794cca8b4114af504f5fb6. Latest consensus state: current_term: 1 leader_uuid: "370f854c39794cca8b4114af504f5fb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370f854c39794cca8b4114af504f5fb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43691 } } }
I20260924 03:26:51.577816 31132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 370f854c39794cca8b4114af504f5fb6 [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:51.577812 31133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 370f854c39794cca8b4114af504f5fb6 [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:51.585590 31124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:51.564131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40051 (local address 127.0.0.1:43691)
0924 03:26:51.564348 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:51.564357 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:51.567587 (+ 3230us) server_negotiation.cc:407] Connection header received
0924 03:26:51.567719 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:51.567723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:51.567792 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:51.567896 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:51.569795 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:51.570644 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:51.571978 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:51.572644 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:51.572770 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:51.575723 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:51.575730 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:51.575735 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:51.575784 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:51.580919 (+ 5135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:51.582097 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:51.582104 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:51.582109 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:51.582331 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:51.582840 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:51.582845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:51.582849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:51.583286 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:51.584693 (+ 1407us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:51.584873 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 03:26:51.585140 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:26:51.588292 31140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:26:51.593015 31140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:26:51.603031 31140 catalog_manager.cc:1347] Generated new cluster ID: 27606690100c4a1faa7d3ec5ba950428
I20260924 03:26:51.603189 31140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:26:51.614346 31140 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:26:51.615708 31140 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:26:51.622999 31140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 370f854c39794cca8b4114af504f5fb6: Generated new TSK 0
I20260924 03:26:51.623925 31140 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:26:51.632936 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 31054
I20260924 03:26:51.634363 31116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:51.726024 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 31054
2026-09-24T03:26:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1948 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:26:51 dist-test-slave-00qs krb5kdc[31153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:26:51 dist-test-slave-00qs krb5kdc[31153](info): set up 2 sockets
Sep 24 03:26:51 dist-test-slave-00qs krb5kdc[31153](info): commencing operation
krb5kdc: starting...
W20260924 03:26:52.764488 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:26:52 dist-test-slave-00qs krb5kdc[31153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:26:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:26:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:26:53.031759 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42239
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:26:53.386690 31166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:53.386986 31166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:53.387121 31166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:53.394809 31166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:26:53.394936 31166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:53.394989 31166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:53.395078 31166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:26:53.395141 31166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:26:53.405491 31166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:53.406792 31166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:53.408093 31166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:53.421079 31172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:53.421020 31171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:53.421000 31174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:53.421809 31166 server_base.cc:1034] running on GCE node
I20260924 03:26:53.422681 31166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:53.424300 31166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:53.425583 31166 hybrid_clock.cc:648] HybridClock initialized: now 1790220413425556 us; error 62 us; skew 500 ppm
I20260924 03:26:53.425971 31166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:26:53 dist-test-slave-00qs krb5kdc[31153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:26:53.429497 31166 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:26:53.431365 31166 webserver.cc:460] Webserver started at http://127.0.0.1:39437/ using document root <none> and password file <none>
I20260924 03:26:53.432201 31166 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:53.432339 31166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:53.432651 31166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:53.435829 31166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "65ca44d4733c49c78be768bb6fec892c"
format_stamp: "Formatted at 2026-09-24 03:26:53 on dist-test-slave-00qs"
server_key: "2dbd94c41369a5e4e7fa08815b659510"
server_key_iv: "c9fb2c29cd310b0a0715c00eaf920a0d"
server_key_version: "encryptionkey@0"
I20260924 03:26:53.436717 31166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "65ca44d4733c49c78be768bb6fec892c"
format_stamp: "Formatted at 2026-09-24 03:26:53 on dist-test-slave-00qs"
server_key: "2dbd94c41369a5e4e7fa08815b659510"
server_key_iv: "c9fb2c29cd310b0a0715c00eaf920a0d"
server_key_version: "encryptionkey@0"
I20260924 03:26:53.442265 31166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 03:26:53.446184 31181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:53.447855 31166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:26:53.448086 31166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "65ca44d4733c49c78be768bb6fec892c"
format_stamp: "Formatted at 2026-09-24 03:26:53 on dist-test-slave-00qs"
server_key: "2dbd94c41369a5e4e7fa08815b659510"
server_key_iv: "c9fb2c29cd310b0a0715c00eaf920a0d"
server_key_version: "encryptionkey@0"
I20260924 03:26:53.448292 31166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:53.465950 31166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:53.470942 31166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:53.471243 31166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:53.490175 31166 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42239
I20260924 03:26:53.490213 31232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42239 every 8 connection(s)
I20260924 03:26:53.491604 31166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:26:53.495388 31233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:53.497558 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 31166
I20260924 03:26:53.497750 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:26:53.498275 10290 external_mini_cluster.cc:1428] Setting key 0797beee39438fcecdd022ab714fbf3a
I20260924 03:26:53.508018 31233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ca44d4733c49c78be768bb6fec892c: Bootstrap starting.
I20260924 03:26:53.511739 31233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65ca44d4733c49c78be768bb6fec892c: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:53.512753 31236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:53.500250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54765 (local address 127.0.0.1:42239)
0924 03:26:53.501094 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:53.501107 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:53.501151 (+ 44us) server_negotiation.cc:407] Connection header received
0924 03:26:53.502914 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:53.502938 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:53.503367 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:53.503722 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:53.504496 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:53.505950 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:53.507795 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:53.508612 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:53.508749 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:53.510164 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:53.510172 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:53.510188 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:53.510767 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:53.510849 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:53.510876 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:53.511071 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:53.511346 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 03:26:53.511641 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":392,"threads_started":1}
I20260924 03:26:53.512957 31233 log.cc:824] T 00000000000000000000000000000000 P 65ca44d4733c49c78be768bb6fec892c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:53.516130 31233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ca44d4733c49c78be768bb6fec892c: No bootstrap required, opened a new log
I20260924 03:26:53.516680 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:26:53.524873 31233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65ca44d4733c49c78be768bb6fec892c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ca44d4733c49c78be768bb6fec892c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42239 } }
I20260924 03:26:53.525383 31233 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65ca44d4733c49c78be768bb6fec892c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:53.525509 31233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65ca44d4733c49c78be768bb6fec892c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65ca44d4733c49c78be768bb6fec892c, State: Initialized, Role: FOLLOWER
I20260924 03:26:53.526217 31233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65ca44d4733c49c78be768bb6fec892c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ca44d4733c49c78be768bb6fec892c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42239 } }
I20260924 03:26:53.526491 31233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65ca44d4733c49c78be768bb6fec892c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:26:53.526630 31233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65ca44d4733c49c78be768bb6fec892c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:26:53.526816 31233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65ca44d4733c49c78be768bb6fec892c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:53.527825 31236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:53.521002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48357 (local address 127.0.0.1:42239)
0924 03:26:53.521202 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:53.521208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:53.521553 (+ 345us) server_negotiation.cc:407] Connection header received
0924 03:26:53.521770 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:53.521775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:53.521850 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:53.521942 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:53.522547 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:53.523324 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:53.525145 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:53.525680 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:53.525787 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:53.527030 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:53.527035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:53.527039 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:53.527058 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:53.527099 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:53.527119 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:53.527193 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:53.527399 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 03:26:53.527491 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 03:26:53.529707 31233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65ca44d4733c49c78be768bb6fec892c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ca44d4733c49c78be768bb6fec892c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42239 } }
I20260924 03:26:53.530184 31233 leader_election.cc:302] T 00000000000000000000000000000000 P 65ca44d4733c49c78be768bb6fec892c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65ca44d4733c49c78be768bb6fec892c; no voters:
I20260924 03:26:53.530735 31233 leader_election.cc:288] T 00000000000000000000000000000000 P 65ca44d4733c49c78be768bb6fec892c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:26:53.530860 31243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65ca44d4733c49c78be768bb6fec892c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:26:53.531520 31243 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65ca44d4733c49c78be768bb6fec892c [term 1 LEADER]: Becoming Leader. State: Replica: 65ca44d4733c49c78be768bb6fec892c, State: Running, Role: LEADER
I20260924 03:26:53.532088 31243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65ca44d4733c49c78be768bb6fec892c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ca44d4733c49c78be768bb6fec892c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42239 } }
I20260924 03:26:53.532799 31233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65ca44d4733c49c78be768bb6fec892c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:26:53.534607 31244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ca44d4733c49c78be768bb6fec892c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65ca44d4733c49c78be768bb6fec892c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ca44d4733c49c78be768bb6fec892c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42239 } } }
I20260924 03:26:53.535148 31245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ca44d4733c49c78be768bb6fec892c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65ca44d4733c49c78be768bb6fec892c. Latest consensus state: current_term: 1 leader_uuid: "65ca44d4733c49c78be768bb6fec892c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ca44d4733c49c78be768bb6fec892c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42239 } } }
I20260924 03:26:53.535409 31244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ca44d4733c49c78be768bb6fec892c [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:53.535470 31245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ca44d4733c49c78be768bb6fec892c [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:53.536760 31248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:26:53.541502 31248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:26:53.551213 31248 catalog_manager.cc:1347] Generated new cluster ID: e449d0bac2a94f839e9882a8c1c74b3a
I20260924 03:26:53.551349 31248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:26:53.576169 31248 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:26:53.578212 31248 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:26:53.588752 31248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65ca44d4733c49c78be768bb6fec892c: Generated new TSK 0
I20260924 03:26:53.589668 31248 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:26:53.605194 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 31166
W20260924 03:26:53.606570 31228 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54765
W20260924 03:26:53.607439 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 31166: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:26:53.607574 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 31166
2026-09-24T03:26:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1881 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:26:53 dist-test-slave-00qs krb5kdc[31264](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:26:53 dist-test-slave-00qs krb5kdc[31264](info): set up 2 sockets
Sep 24 03:26:53 dist-test-slave-00qs krb5kdc[31264](info): commencing operation
krb5kdc: starting...
W20260924 03:26:54.636463 10290 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:26:54 dist-test-slave-00qs krb5kdc[31264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:26:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:26:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:26:54.895407 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36385
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:26:55.270238 31277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:55.270695 31277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:55.270838 31277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:55.279153 31277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:26:55.279328 31277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:55.279382 31277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:55.279481 31277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:26:55.279547 31277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:26:55.291332 31277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:55.293145 31277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:55.294932 31277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:55.309850 31282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:55.309811 31283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:55.309892 31285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:55.310820 31277 server_base.cc:1034] running on GCE node
I20260924 03:26:55.311863 31277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:55.313890 31277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:55.315124 31277 hybrid_clock.cc:648] HybridClock initialized: now 1790220415315071 us; error 56 us; skew 500 ppm
I20260924 03:26:55.315519 31277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:26:55 dist-test-slave-00qs krb5kdc[31264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:26:55.319284 31277 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:26:55.321276 31277 webserver.cc:460] Webserver started at http://127.0.0.1:40019/ using document root <none> and password file <none>
I20260924 03:26:55.322141 31277 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:55.322263 31277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:55.322613 31277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:55.326380 31277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "11c7f2d9586b42519139cde1d49f8fcb"
format_stamp: "Formatted at 2026-09-24 03:26:55 on dist-test-slave-00qs"
server_key: "baeaeeb37dcc45f19c06ca1471f4362d"
server_key_iv: "cba3c3cb80e6f2d42037663ad7dcfd50"
server_key_version: "encryptionkey@0"
I20260924 03:26:55.327369 31277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "11c7f2d9586b42519139cde1d49f8fcb"
format_stamp: "Formatted at 2026-09-24 03:26:55 on dist-test-slave-00qs"
server_key: "baeaeeb37dcc45f19c06ca1471f4362d"
server_key_iv: "cba3c3cb80e6f2d42037663ad7dcfd50"
server_key_version: "encryptionkey@0"
I20260924 03:26:55.334183 31277 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260924 03:26:55.339113 31292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:55.341150 31277 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 03:26:55.341375 31277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "11c7f2d9586b42519139cde1d49f8fcb"
format_stamp: "Formatted at 2026-09-24 03:26:55 on dist-test-slave-00qs"
server_key: "baeaeeb37dcc45f19c06ca1471f4362d"
server_key_iv: "cba3c3cb80e6f2d42037663ad7dcfd50"
server_key_version: "encryptionkey@0"
I20260924 03:26:55.341647 31277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:55.349014 31277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:55.354593 31277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:55.354915 31277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:55.376431 31277 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36385
I20260924 03:26:55.376466 31343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36385 every 8 connection(s)
I20260924 03:26:55.378561 31277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:26:55.383517 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 31277
I20260924 03:26:55.383733 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:26:55.384352 10290 external_mini_cluster.cc:1428] Setting key 90c0c49957e66fdbb62ce03e5bde1c07
I20260924 03:26:55.384531 31344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:55.394642 31347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:55.386749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40447 (local address 127.0.0.1:36385)
0924 03:26:55.387702 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:55.387716 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:55.387769 (+ 53us) server_negotiation.cc:407] Connection header received
0924 03:26:55.390076 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:55.390104 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:55.390626 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:55.391071 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:55.392396 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:55.392404 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:55.392419 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:55.392446 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:55.392527 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:55.392553 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:55.392757 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:55.392893 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 03:26:55.393150 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":400,"threads_started":1}
I20260924 03:26:55.399585 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:26:55.400851 31344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c7f2d9586b42519139cde1d49f8fcb: Bootstrap starting.
I20260924 03:26:55.404963 31344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11c7f2d9586b42519139cde1d49f8fcb: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:55.406375 31344 log.cc:824] T 00000000000000000000000000000000 P 11c7f2d9586b42519139cde1d49f8fcb: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:55.407855 31347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:55.404873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56307 (local address 127.0.0.1:36385)
0924 03:26:55.405138 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:55.405147 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:55.405673 (+ 526us) server_negotiation.cc:407] Connection header received
0924 03:26:55.405880 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:55.405885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:55.405952 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:55.406056 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:55.407140 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:55.407145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:55.407148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:55.407167 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:55.407219 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:55.407239 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:55.407351 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:55.407468 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 03:26:55.407548 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 03:26:55.410244 31344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c7f2d9586b42519139cde1d49f8fcb: No bootstrap required, opened a new log
I20260924 03:26:55.419242 31344 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11c7f2d9586b42519139cde1d49f8fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c7f2d9586b42519139cde1d49f8fcb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36385 } }
I20260924 03:26:55.419811 31344 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11c7f2d9586b42519139cde1d49f8fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:55.419927 31344 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11c7f2d9586b42519139cde1d49f8fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c7f2d9586b42519139cde1d49f8fcb, State: Initialized, Role: FOLLOWER
I20260924 03:26:55.420953 31344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11c7f2d9586b42519139cde1d49f8fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c7f2d9586b42519139cde1d49f8fcb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36385 } }
I20260924 03:26:55.421305 31344 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11c7f2d9586b42519139cde1d49f8fcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:26:55.421473 31344 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11c7f2d9586b42519139cde1d49f8fcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:26:55.421658 31344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11c7f2d9586b42519139cde1d49f8fcb [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:55.424558 31344 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11c7f2d9586b42519139cde1d49f8fcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c7f2d9586b42519139cde1d49f8fcb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36385 } }
I20260924 03:26:55.425318 31344 leader_election.cc:302] T 00000000000000000000000000000000 P 11c7f2d9586b42519139cde1d49f8fcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11c7f2d9586b42519139cde1d49f8fcb; no voters:
I20260924 03:26:55.426017 31344 leader_election.cc:288] T 00000000000000000000000000000000 P 11c7f2d9586b42519139cde1d49f8fcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:26:55.426236 31354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11c7f2d9586b42519139cde1d49f8fcb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:26:55.427002 31354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11c7f2d9586b42519139cde1d49f8fcb [term 1 LEADER]: Becoming Leader. State: Replica: 11c7f2d9586b42519139cde1d49f8fcb, State: Running, Role: LEADER
I20260924 03:26:55.427650 31354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11c7f2d9586b42519139cde1d49f8fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c7f2d9586b42519139cde1d49f8fcb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36385 } }
I20260924 03:26:55.428668 31344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11c7f2d9586b42519139cde1d49f8fcb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:26:55.431375 31355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11c7f2d9586b42519139cde1d49f8fcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11c7f2d9586b42519139cde1d49f8fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c7f2d9586b42519139cde1d49f8fcb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36385 } } }
I20260924 03:26:55.432154 31355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11c7f2d9586b42519139cde1d49f8fcb [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:55.431809 31356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11c7f2d9586b42519139cde1d49f8fcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11c7f2d9586b42519139cde1d49f8fcb. Latest consensus state: current_term: 1 leader_uuid: "11c7f2d9586b42519139cde1d49f8fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c7f2d9586b42519139cde1d49f8fcb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36385 } } }
I20260924 03:26:55.432310 31356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11c7f2d9586b42519139cde1d49f8fcb [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:55.434141 31363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:26:55.440302 31363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:26:55.452530 31363 catalog_manager.cc:1347] Generated new cluster ID: d34bf2337039499fbd4cdc3b507c05de
I20260924 03:26:55.452736 31363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:26:55.468257 31363 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:26:55.468922 31363 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:26:55.479858 31363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11c7f2d9586b42519139cde1d49f8fcb: Generated new TSK 0
I20260924 03:26:55.481462 31363 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:26:55.499337 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 31277
W20260924 03:26:55.500919 31339 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40447
W20260924 03:26:55.501968 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 31277: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:26:55.502233 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 31277
2026-09-24T03:26:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1898 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:26:55 dist-test-slave-00qs krb5kdc[31375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:26:55 dist-test-slave-00qs krb5kdc[31375](info): set up 2 sockets
Sep 24 03:26:55 dist-test-slave-00qs krb5kdc[31375](info): commencing operation
krb5kdc: starting...
W20260924 03:26:56.573069 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:26:56 dist-test-slave-00qs krb5kdc[31375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:26:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:26:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:26:56.896035 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45049
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:26:57.289605 31388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:57.289942 31388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:57.290063 31388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:57.297847 31388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:26:57.298007 31388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:57.298084 31388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:57.298164 31388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:26:57.298228 31388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:26:57.308861 31388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:57.310321 31388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:57.311733 31388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:57.324388 31396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:57.324357 31393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:57.324357 31394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:57.325480 31388 server_base.cc:1034] running on GCE node
I20260924 03:26:57.326321 31388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:57.327943 31388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:57.329236 31388 hybrid_clock.cc:648] HybridClock initialized: now 1790220417329178 us; error 47 us; skew 500 ppm
I20260924 03:26:57.329710 31388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:26:57 dist-test-slave-00qs krb5kdc[31375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:26:57.333360 31388 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:26:57.335409 31388 webserver.cc:460] Webserver started at http://127.0.0.1:33101/ using document root <none> and password file <none>
I20260924 03:26:57.336288 31388 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:57.336421 31388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:57.336746 31388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:57.340018 31388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "8c2d9dc9e2404c309a48a095de01c4ab"
format_stamp: "Formatted at 2026-09-24 03:26:57 on dist-test-slave-00qs"
server_key: "3d5a14f4bc2a2bedbb38794062039b35"
server_key_iv: "1f63470eee4208a389cc5cf9ab53ca23"
server_key_version: "encryptionkey@0"
I20260924 03:26:57.340898 31388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "8c2d9dc9e2404c309a48a095de01c4ab"
format_stamp: "Formatted at 2026-09-24 03:26:57 on dist-test-slave-00qs"
server_key: "3d5a14f4bc2a2bedbb38794062039b35"
server_key_iv: "1f63470eee4208a389cc5cf9ab53ca23"
server_key_version: "encryptionkey@0"
I20260924 03:26:57.346597 31388 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:26:57.350435 31403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:57.352134 31388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:26:57.352347 31388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "8c2d9dc9e2404c309a48a095de01c4ab"
format_stamp: "Formatted at 2026-09-24 03:26:57 on dist-test-slave-00qs"
server_key: "3d5a14f4bc2a2bedbb38794062039b35"
server_key_iv: "1f63470eee4208a389cc5cf9ab53ca23"
server_key_version: "encryptionkey@0"
I20260924 03:26:57.352560 31388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:57.382172 31388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:57.386957 31388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:57.387255 31388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:57.407156 31388 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45049
I20260924 03:26:57.407192 31454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45049 every 8 connection(s)
I20260924 03:26:57.408653 31388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:26:57.409730 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 31388
I20260924 03:26:57.409950 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:26:57.410604 10290 external_mini_cluster.cc:1428] Setting key 17703ede960001c79112536a4829b11f
I20260924 03:26:57.413767 31455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 03:26:57 dist-test-slave-00qs krb5kdc[31375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 03:26:57.427727 31455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c2d9dc9e2404c309a48a095de01c4ab: Bootstrap starting.
I20260924 03:26:57.432330 31455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c2d9dc9e2404c309a48a095de01c4ab: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:57.433611 31455 log.cc:824] T 00000000000000000000000000000000 P 8c2d9dc9e2404c309a48a095de01c4ab: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:57.437397 31455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c2d9dc9e2404c309a48a095de01c4ab: No bootstrap required, opened a new log
I20260924 03:26:57.437592 31458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:57.412843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.212:59991 (local address 127.0.0.1:45049)
0924 03:26:57.413674 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:57.413686 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:57.413731 (+ 45us) server_negotiation.cc:407] Connection header received
0924 03:26:57.415396 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:57.415423 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:57.415858 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:57.416297 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:57.418131 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:57.419626 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:57.421760 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:57.422552 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:57.422713 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:57.428099 (+ 5386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:57.428114 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:57.428138 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:57.428194 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:57.432225 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:57.433542 (+ 1317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:57.433552 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:57.433562 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:57.433818 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:57.434391 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:57.434396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:57.434399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:57.434993 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:57.435340 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:57.435794 (+ 454us) server_negotiation.cc:299] Negotiation successful
0924 03:26:57.436199 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":362,"threads_started":1}
I20260924 03:26:57.442839 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:26:57.447912 31455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c2d9dc9e2404c309a48a095de01c4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2d9dc9e2404c309a48a095de01c4ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45049 } }
I20260924 03:26:57.448483 31455 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c2d9dc9e2404c309a48a095de01c4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:57.448607 31455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c2d9dc9e2404c309a48a095de01c4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c2d9dc9e2404c309a48a095de01c4ab, State: Initialized, Role: FOLLOWER
I20260924 03:26:57.449355 31455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c2d9dc9e2404c309a48a095de01c4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2d9dc9e2404c309a48a095de01c4ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45049 } }
I20260924 03:26:57.449676 31455 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c2d9dc9e2404c309a48a095de01c4ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:26:57.449826 31455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c2d9dc9e2404c309a48a095de01c4ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:26:57.449998 31455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c2d9dc9e2404c309a48a095de01c4ab [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:57.452687 31455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c2d9dc9e2404c309a48a095de01c4ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2d9dc9e2404c309a48a095de01c4ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45049 } }
I20260924 03:26:57.453362 31455 leader_election.cc:302] T 00000000000000000000000000000000 P 8c2d9dc9e2404c309a48a095de01c4ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c2d9dc9e2404c309a48a095de01c4ab; no voters:
I20260924 03:26:57.454077 31455 leader_election.cc:288] T 00000000000000000000000000000000 P 8c2d9dc9e2404c309a48a095de01c4ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:26:57.454202 31465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c2d9dc9e2404c309a48a095de01c4ab [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:26:57.456203 31465 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c2d9dc9e2404c309a48a095de01c4ab [term 1 LEADER]: Becoming Leader. State: Replica: 8c2d9dc9e2404c309a48a095de01c4ab, State: Running, Role: LEADER
I20260924 03:26:57.456836 31465 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c2d9dc9e2404c309a48a095de01c4ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2d9dc9e2404c309a48a095de01c4ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45049 } }
I20260924 03:26:57.457227 31455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c2d9dc9e2404c309a48a095de01c4ab [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:26:57.465170 31458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:57.449273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.212:40579 (local address 127.0.0.1:45049)
0924 03:26:57.449519 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:57.449526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:57.450095 (+ 569us) server_negotiation.cc:407] Connection header received
0924 03:26:57.450345 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:57.450351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:57.450451 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:57.450601 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:57.452231 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:57.453323 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:57.454927 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:57.455622 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:57.455741 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:57.458861 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:57.458867 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:57.458870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:26:57.458907 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:57.461809 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:57.462925 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:57.462929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:57.462931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:57.463066 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:26:57.463515 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:26:57.463519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:26:57.463521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:26:57.463829 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:57.464153 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:57.464563 (+ 410us) server_negotiation.cc:299] Negotiation successful
0924 03:26:57.464762 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 03:26:57.466284 31466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c2d9dc9e2404c309a48a095de01c4ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c2d9dc9e2404c309a48a095de01c4ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2d9dc9e2404c309a48a095de01c4ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45049 } } }
I20260924 03:26:57.466722 31466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c2d9dc9e2404c309a48a095de01c4ab [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:57.467262 31467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c2d9dc9e2404c309a48a095de01c4ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c2d9dc9e2404c309a48a095de01c4ab. Latest consensus state: current_term: 1 leader_uuid: "8c2d9dc9e2404c309a48a095de01c4ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2d9dc9e2404c309a48a095de01c4ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45049 } } }
I20260924 03:26:57.467559 31473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:26:57.467612 31467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c2d9dc9e2404c309a48a095de01c4ab [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:57.473825 31473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:26:57.486248 31473 catalog_manager.cc:1347] Generated new cluster ID: c3102eb75c5d44b3b439dea751a4d3be
I20260924 03:26:57.486392 31473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:26:57.504498 31473 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:26:57.506471 31473 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:26:57.514762 31473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c2d9dc9e2404c309a48a095de01c4ab: Generated new TSK 0
I20260924 03:26:57.515986 31473 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:26:57.528976 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 31388
I20260924 03:26:57.530982 31450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:26:57.629431 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 31388
W20260924 03:26:57.644276 31373 connection.cc:535] client connection to 127.0.0.1:45049 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45049): Connection reset by peer (error 104)
2026-09-24T03:26:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2123 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:26:57 dist-test-slave-00qs krb5kdc[31488](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:26:57 dist-test-slave-00qs krb5kdc[31488](info): set up 2 sockets
Sep 24 03:26:57 dist-test-slave-00qs krb5kdc[31488](info): commencing operation
krb5kdc: starting...
W20260924 03:26:58.676602 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:26:58 dist-test-slave-00qs krb5kdc[31488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:26:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:26:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:26:58.944038 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45897
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:26:59.298993 31501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:26:59.299310 31501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:26:59.299440 31501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:26:59.307385 31501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:26:59.307541 31501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:26:59.307593 31501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:26:59.307680 31501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:26:59.307734 31501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:26:59.318172 31501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:26:59.319548 31501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:26:59.320911 31501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:26:59.334390 31509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:59.334494 31506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:26:59.334466 31507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:26:59.335260 31501 server_base.cc:1034] running on GCE node
I20260924 03:26:59.336199 31501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:26:59.337841 31501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:26:59.339259 31501 hybrid_clock.cc:648] HybridClock initialized: now 1790220419339122 us; error 107 us; skew 500 ppm
I20260924 03:26:59.340003 31501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:26:59 dist-test-slave-00qs krb5kdc[31488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:26:59.356297 31501 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:26:59.358114 31501 webserver.cc:460] Webserver started at http://127.0.0.1:33661/ using document root <none> and password file <none>
I20260924 03:26:59.358868 31501 fs_manager.cc:359] Metadata directory not provided
I20260924 03:26:59.359014 31501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:26:59.359337 31501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:26:59.362624 31501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "72d1082c8b004f63bad225317fd9c3fc"
format_stamp: "Formatted at 2026-09-24 03:26:59 on dist-test-slave-00qs"
server_key: "d23bace875450b68e332690bbd882e07"
server_key_iv: "59de8e9a485c3b1db7a271b30294da52"
server_key_version: "encryptionkey@0"
I20260924 03:26:59.363478 31501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "72d1082c8b004f63bad225317fd9c3fc"
format_stamp: "Formatted at 2026-09-24 03:26:59 on dist-test-slave-00qs"
server_key: "d23bace875450b68e332690bbd882e07"
server_key_iv: "59de8e9a485c3b1db7a271b30294da52"
server_key_version: "encryptionkey@0"
I20260924 03:26:59.369418 31501 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 03:26:59.373721 31516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:26:59.375895 31501 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:26:59.376164 31501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "72d1082c8b004f63bad225317fd9c3fc"
format_stamp: "Formatted at 2026-09-24 03:26:59 on dist-test-slave-00qs"
server_key: "d23bace875450b68e332690bbd882e07"
server_key_iv: "59de8e9a485c3b1db7a271b30294da52"
server_key_version: "encryptionkey@0"
I20260924 03:26:59.376389 31501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:26:59.388923 31501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:26:59.393344 31501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:26:59.393649 31501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:26:59.413020 31501 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45897
I20260924 03:26:59.413172 31567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45897 every 8 connection(s)
I20260924 03:26:59.415117 31501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:26:59.419824 31568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:26:59.421352 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 31501
I20260924 03:26:59.421698 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:26:59.422421 10290 external_mini_cluster.cc:1428] Setting key f81186c25f6f2142c918432197a2042d
I20260924 03:26:59.436558 31568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d1082c8b004f63bad225317fd9c3fc: Bootstrap starting.
I20260924 03:26:59.438251 31571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:59.424420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.212:53993 (local address 127.0.0.1:45897)
0924 03:26:59.425245 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:59.425261 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:59.425306 (+ 45us) server_negotiation.cc:407] Connection header received
0924 03:26:59.427499 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:59.427534 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:59.428140 (+ 606us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:59.428545 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:59.429201 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:59.430548 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:59.432582 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:59.433713 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:59.433891 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:59.435674 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:59.435683 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:59.435702 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:59.436378 (+ 676us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:59.436473 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:59.436511 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:59.436647 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:59.436953 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 03:26:59.437167 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":351,"threads_started":1}
I20260924 03:26:59.439874 31568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72d1082c8b004f63bad225317fd9c3fc: Neither blocks nor log segments found. Creating new log.
I20260924 03:26:59.441042 31568 log.cc:824] T 00000000000000000000000000000000 P 72d1082c8b004f63bad225317fd9c3fc: Log is configured to *not* fsync() on all Append() calls
I20260924 03:26:59.442423 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:26:59.444145 31568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d1082c8b004f63bad225317fd9c3fc: No bootstrap required, opened a new log
I20260924 03:26:59.453632 31568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72d1082c8b004f63bad225317fd9c3fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d1082c8b004f63bad225317fd9c3fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45897 } }
I20260924 03:26:59.454165 31568 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72d1082c8b004f63bad225317fd9c3fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:26:59.454269 31568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72d1082c8b004f63bad225317fd9c3fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d1082c8b004f63bad225317fd9c3fc, State: Initialized, Role: FOLLOWER
I20260924 03:26:59.454999 31568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72d1082c8b004f63bad225317fd9c3fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d1082c8b004f63bad225317fd9c3fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45897 } }
I20260924 03:26:59.455323 31568 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72d1082c8b004f63bad225317fd9c3fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:26:59.455377 31571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:26:59.447029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.212:35085 (local address 127.0.0.1:45897)
0924 03:26:59.447233 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 03:26:59.447240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:26:59.447526 (+ 286us) server_negotiation.cc:407] Connection header received
0924 03:26:59.447687 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:26:59.447697 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:26:59.447791 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:26:59.447911 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:26:59.448881 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:59.450085 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:59.451927 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:26:59.452684 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:26:59.452861 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:26:59.454516 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:26:59.454523 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:26:59.454527 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:26:59.454553 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:26:59.454628 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:26:59.454651 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:26:59.454774 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 03:26:59.454998 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 03:26:59.455065 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 03:26:59.455484 31568 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d1082c8b004f63bad225317fd9c3fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:26:59.455688 31568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72d1082c8b004f63bad225317fd9c3fc [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:26:59.458635 31568 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d1082c8b004f63bad225317fd9c3fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d1082c8b004f63bad225317fd9c3fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45897 } }
I20260924 03:26:59.459183 31568 leader_election.cc:302] T 00000000000000000000000000000000 P 72d1082c8b004f63bad225317fd9c3fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72d1082c8b004f63bad225317fd9c3fc; no voters:
I20260924 03:26:59.459862 31568 leader_election.cc:288] T 00000000000000000000000000000000 P 72d1082c8b004f63bad225317fd9c3fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:26:59.460181 31578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72d1082c8b004f63bad225317fd9c3fc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:26:59.461530 31578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72d1082c8b004f63bad225317fd9c3fc [term 1 LEADER]: Becoming Leader. State: Replica: 72d1082c8b004f63bad225317fd9c3fc, State: Running, Role: LEADER
I20260924 03:26:59.462097 31578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72d1082c8b004f63bad225317fd9c3fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d1082c8b004f63bad225317fd9c3fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45897 } }
I20260924 03:26:59.462622 31568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72d1082c8b004f63bad225317fd9c3fc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:26:59.464726 31580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d1082c8b004f63bad225317fd9c3fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72d1082c8b004f63bad225317fd9c3fc. Latest consensus state: current_term: 1 leader_uuid: "72d1082c8b004f63bad225317fd9c3fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d1082c8b004f63bad225317fd9c3fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45897 } } }
I20260924 03:26:59.465060 31580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d1082c8b004f63bad225317fd9c3fc [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:59.465608 31579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d1082c8b004f63bad225317fd9c3fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72d1082c8b004f63bad225317fd9c3fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d1082c8b004f63bad225317fd9c3fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45897 } } }
I20260924 03:26:59.465921 31583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:26:59.465967 31579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d1082c8b004f63bad225317fd9c3fc [sys.catalog]: This master's current role is: LEADER
I20260924 03:26:59.471752 31583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:26:59.482316 31583 catalog_manager.cc:1347] Generated new cluster ID: 36d1a77ebd2943188eeb8024bf914cdd
I20260924 03:26:59.482477 31583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:26:59.494670 31583 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:26:59.496143 31583 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:26:59.505584 31583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72d1082c8b004f63bad225317fd9c3fc: Generated new TSK 0
I20260924 03:26:59.506567 31583 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:26:59.513108 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 31501
W20260924 03:26:59.514295 31563 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.212:53993
W20260924 03:26:59.515056 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 31501: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:26:59.515219 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 31501
W20260924 03:26:59.529006 31486 connection.cc:535] client connection to 127.0.0.1:45897 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45897): Connection reset by peer (error 104)
2026-09-24T03:26:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1885 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:26:59 dist-test-slave-00qs krb5kdc[31599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:26:59 dist-test-slave-00qs krb5kdc[31599](info): set up 2 sockets
Sep 24 03:26:59 dist-test-slave-00qs krb5kdc[31599](info): commencing operation
krb5kdc: starting...
W20260924 03:27:00.556422 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:27:00 dist-test-slave-00qs krb5kdc[31599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:27:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:27:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:27:00.819701 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42147
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:27:01.158684 31612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:27:01.159013 31612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:27:01.159109 31612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:27:01.166553 31612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:27:01.166682 31612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:27:01.166728 31612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:27:01.166786 31612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:27:01.166829 31612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:27:01.177516 31612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:27:01.178814 31612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:27:01.180136 31612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:27:01.192245 31617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:01.192212 31618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:01.192174 31620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:27:01.193094 31612 server_base.cc:1034] running on GCE node
I20260924 03:27:01.193913 31612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:27:01.195590 31612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:27:01.196857 31612 hybrid_clock.cc:648] HybridClock initialized: now 1790220421196796 us; error 45 us; skew 500 ppm
I20260924 03:27:01.197263 31612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:27:01 dist-test-slave-00qs krb5kdc[31599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:27:01.200932 31612 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:27:01.202652 31612 webserver.cc:460] Webserver started at http://127.0.0.1:41707/ using document root <none> and password file <none>
I20260924 03:27:01.203358 31612 fs_manager.cc:359] Metadata directory not provided
I20260924 03:27:01.203487 31612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:27:01.203781 31612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:27:01.206955 31612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "24a0133e348f4bbc85dbe776cff1ff1e"
format_stamp: "Formatted at 2026-09-24 03:27:01 on dist-test-slave-00qs"
server_key: "11c66873d0b415ec64d0c41812996646"
server_key_iv: "bc4b40f177a538d159ed00870126043e"
server_key_version: "encryptionkey@0"
I20260924 03:27:01.207827 31612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "24a0133e348f4bbc85dbe776cff1ff1e"
format_stamp: "Formatted at 2026-09-24 03:27:01 on dist-test-slave-00qs"
server_key: "11c66873d0b415ec64d0c41812996646"
server_key_iv: "bc4b40f177a538d159ed00870126043e"
server_key_version: "encryptionkey@0"
I20260924 03:27:01.213517 31612 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 03:27:01.217486 31627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:27:01.218964 31612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:27:01.219159 31612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "24a0133e348f4bbc85dbe776cff1ff1e"
format_stamp: "Formatted at 2026-09-24 03:27:01 on dist-test-slave-00qs"
server_key: "11c66873d0b415ec64d0c41812996646"
server_key_iv: "bc4b40f177a538d159ed00870126043e"
server_key_version: "encryptionkey@0"
I20260924 03:27:01.219370 31612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:27:01.225540 31612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:27:01.230028 31612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:27:01.230347 31612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:27:01.248314 31612 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42147
I20260924 03:27:01.248327 31678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42147 every 8 connection(s)
I20260924 03:27:01.249763 31612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:27:01.253638 31679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:27:01.255828 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 31612
I20260924 03:27:01.256064 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:27:01.256608 10290 external_mini_cluster.cc:1428] Setting key 3bec4259fa9e3fc64efaee3238b34c6c
I20260924 03:27:01.265417 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:01.258698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.212:52229 (local address 127.0.0.1:42147)
0924 03:27:01.259792 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:01.259804 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:01.259859 (+ 55us) server_negotiation.cc:407] Connection header received
0924 03:27:01.261638 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:01.261664 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:01.262066 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:01.262491 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:01.263628 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:01.263636 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:01.263653 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:27:01.263669 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:01.263739 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:27:01.263761 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:01.264046 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:01.264193 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 03:27:01.264407 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":854,"thread_start_us":602,"threads_started":1}
I20260924 03:27:01.266373 31679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24a0133e348f4bbc85dbe776cff1ff1e: Bootstrap starting.
I20260924 03:27:01.269580 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:27:01.269836 31679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24a0133e348f4bbc85dbe776cff1ff1e: Neither blocks nor log segments found. Creating new log.
I20260924 03:27:01.270915 31679 log.cc:824] T 00000000000000000000000000000000 P 24a0133e348f4bbc85dbe776cff1ff1e: Log is configured to *not* fsync() on all Append() calls
I20260924 03:27:01.273852 31679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24a0133e348f4bbc85dbe776cff1ff1e: No bootstrap required, opened a new log
I20260924 03:27:01.277343 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:01.274517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.212:41445 (local address 127.0.0.1:42147)
0924 03:27:01.274731 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:01.274738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:01.275196 (+ 458us) server_negotiation.cc:407] Connection header received
0924 03:27:01.275338 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:01.275343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:01.275413 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:01.275540 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:01.276481 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:01.276488 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:01.276493 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:27:01.276517 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:01.276571 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:27:01.276594 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:01.276724 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:01.276894 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 03:27:01.276972 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 03:27:01.282971 31679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24a0133e348f4bbc85dbe776cff1ff1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a0133e348f4bbc85dbe776cff1ff1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42147 } }
I20260924 03:27:01.283372 31679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24a0133e348f4bbc85dbe776cff1ff1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:27:01.283438 31679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24a0133e348f4bbc85dbe776cff1ff1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a0133e348f4bbc85dbe776cff1ff1e, State: Initialized, Role: FOLLOWER
I20260924 03:27:01.284044 31679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24a0133e348f4bbc85dbe776cff1ff1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a0133e348f4bbc85dbe776cff1ff1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42147 } }
I20260924 03:27:01.284276 31679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24a0133e348f4bbc85dbe776cff1ff1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:27:01.284415 31679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24a0133e348f4bbc85dbe776cff1ff1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:27:01.284569 31679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24a0133e348f4bbc85dbe776cff1ff1e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:27:01.286824 31679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24a0133e348f4bbc85dbe776cff1ff1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a0133e348f4bbc85dbe776cff1ff1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42147 } }
I20260924 03:27:01.287400 31679 leader_election.cc:302] T 00000000000000000000000000000000 P 24a0133e348f4bbc85dbe776cff1ff1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24a0133e348f4bbc85dbe776cff1ff1e; no voters:
I20260924 03:27:01.288024 31679 leader_election.cc:288] T 00000000000000000000000000000000 P 24a0133e348f4bbc85dbe776cff1ff1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:27:01.288159 31689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24a0133e348f4bbc85dbe776cff1ff1e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:27:01.288780 31689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24a0133e348f4bbc85dbe776cff1ff1e [term 1 LEADER]: Becoming Leader. State: Replica: 24a0133e348f4bbc85dbe776cff1ff1e, State: Running, Role: LEADER
I20260924 03:27:01.289321 31689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24a0133e348f4bbc85dbe776cff1ff1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a0133e348f4bbc85dbe776cff1ff1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42147 } }
I20260924 03:27:01.290040 31679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24a0133e348f4bbc85dbe776cff1ff1e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:27:01.292096 31691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24a0133e348f4bbc85dbe776cff1ff1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24a0133e348f4bbc85dbe776cff1ff1e. Latest consensus state: current_term: 1 leader_uuid: "24a0133e348f4bbc85dbe776cff1ff1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a0133e348f4bbc85dbe776cff1ff1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42147 } } }
I20260924 03:27:01.292639 31691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24a0133e348f4bbc85dbe776cff1ff1e [sys.catalog]: This master's current role is: LEADER
I20260924 03:27:01.292035 31690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24a0133e348f4bbc85dbe776cff1ff1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24a0133e348f4bbc85dbe776cff1ff1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a0133e348f4bbc85dbe776cff1ff1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42147 } } }
I20260924 03:27:01.294323 31690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24a0133e348f4bbc85dbe776cff1ff1e [sys.catalog]: This master's current role is: LEADER
I20260924 03:27:01.295353 31696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:27:01.301815 31696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:27:01.312805 31696 catalog_manager.cc:1347] Generated new cluster ID: 14869479568d45e2a8256ad5559085c0
I20260924 03:27:01.312938 31696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:27:01.342361 31696 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:27:01.342792 31696 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:27:01.350942 31696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24a0133e348f4bbc85dbe776cff1ff1e: Generated new TSK 0
I20260924 03:27:01.351886 31696 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:27:01.362774 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 31612
W20260924 03:27:01.365134 31674 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.212:52229
W20260924 03:27:01.365970 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 31612: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:27:01.366139 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 31612
2026-09-24T03:27:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1851 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:27:01 dist-test-slave-00qs krb5kdc[31710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:27:01 dist-test-slave-00qs krb5kdc[31710](info): set up 2 sockets
Sep 24 03:27:01 dist-test-slave-00qs krb5kdc[31710](info): commencing operation
krb5kdc: starting...
W20260924 03:27:02.424582 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:27:02 dist-test-slave-00qs krb5kdc[31710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:27:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:27:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:27:02.692416 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44907
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:27:03.031441 31723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:27:03.031752 31723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:27:03.031847 31723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:27:03.039294 31723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:27:03.039435 31723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:27:03.039482 31723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:27:03.039541 31723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:27:03.039584 31723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:27:03.049896 31723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44907
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:27:03.051229 31723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:27:03.052493 31723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:27:03.064093 31728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:03.064148 31729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:03.064127 31731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:27:03.064833 31723 server_base.cc:1034] running on GCE node
I20260924 03:27:03.065706 31723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:27:03.067216 31723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:27:03.068444 31723 hybrid_clock.cc:648] HybridClock initialized: now 1790220423068395 us; error 34 us; skew 500 ppm
I20260924 03:27:03.068805 31723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:27:03 dist-test-slave-00qs krb5kdc[31710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:27:03.072024 31723 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:27:03.073578 31723 webserver.cc:460] Webserver started at http://127.0.0.1:46499/ using document root <none> and password file <none>
I20260924 03:27:03.074241 31723 fs_manager.cc:359] Metadata directory not provided
I20260924 03:27:03.074348 31723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:27:03.074604 31723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:27:03.077661 31723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "467577471b9c4e3ea7d633f414649bed"
format_stamp: "Formatted at 2026-09-24 03:27:03 on dist-test-slave-00qs"
server_key: "f1bc8ea3db9bf4bdaad7ddec95322973"
server_key_iv: "b262639bc27c90b033f2e96befc9d5fe"
server_key_version: "encryptionkey@0"
I20260924 03:27:03.078423 31723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "467577471b9c4e3ea7d633f414649bed"
format_stamp: "Formatted at 2026-09-24 03:27:03 on dist-test-slave-00qs"
server_key: "f1bc8ea3db9bf4bdaad7ddec95322973"
server_key_iv: "b262639bc27c90b033f2e96befc9d5fe"
server_key_version: "encryptionkey@0"
I20260924 03:27:03.083776 31723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:27:03.087540 31738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:27:03.088977 31723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:27:03.089159 31723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "467577471b9c4e3ea7d633f414649bed"
format_stamp: "Formatted at 2026-09-24 03:27:03 on dist-test-slave-00qs"
server_key: "f1bc8ea3db9bf4bdaad7ddec95322973"
server_key_iv: "b262639bc27c90b033f2e96befc9d5fe"
server_key_version: "encryptionkey@0"
I20260924 03:27:03.089360 31723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:27:03.106734 31723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:27:03.111503 31723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:27:03.111794 31723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:27:03.130558 31723 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44907
I20260924 03:27:03.130613 31789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44907 every 8 connection(s)
I20260924 03:27:03.131985 31723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:27:03.135797 31790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:27:03.138489 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 31723
I20260924 03:27:03.138697 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:27:03.139240 10290 external_mini_cluster.cc:1428] Setting key db96a489f1b1de9780fdf7c6bf180359
I20260924 03:27:03.149349 31790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 467577471b9c4e3ea7d633f414649bed: Bootstrap starting.
I20260924 03:27:03.152696 31790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 467577471b9c4e3ea7d633f414649bed: Neither blocks nor log segments found. Creating new log.
I20260924 03:27:03.153751 31790 log.cc:824] T 00000000000000000000000000000000 P 467577471b9c4e3ea7d633f414649bed: Log is configured to *not* fsync() on all Append() calls
Sep 24 03:27:03 dist-test-slave-00qs krb5kdc[31710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 03:27:03.156909 31790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 467577471b9c4e3ea7d633f414649bed: No bootstrap required, opened a new log
I20260924 03:27:03.165575 31793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:03.141854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.212:60821 (local address 127.0.0.1:44907)
0924 03:27:03.142871 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:03.142887 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:03.142946 (+ 59us) server_negotiation.cc:407] Connection header received
0924 03:27:03.145098 (+ 2152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:03.145132 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:03.145653 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:03.146070 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:03.148038 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:03.149411 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:03.150808 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:03.151567 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:03.151760 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:03.156089 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:03.156099 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:03.156115 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:03.156161 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:03.159598 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:03.160663 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:03.160670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:03.160677 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:03.160863 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:03.161341 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:03.161346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:03.161349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:03.162149 (+ 800us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:03.162522 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:03.163581 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0924 03:27:03.164033 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":460,"threads_started":1}
I20260924 03:27:03.167171 31790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 467577471b9c4e3ea7d633f414649bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467577471b9c4e3ea7d633f414649bed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44907 } }
I20260924 03:27:03.167573 31790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 467577471b9c4e3ea7d633f414649bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:27:03.167691 31790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 467577471b9c4e3ea7d633f414649bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 467577471b9c4e3ea7d633f414649bed, State: Initialized, Role: FOLLOWER
I20260924 03:27:03.168358 31790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 467577471b9c4e3ea7d633f414649bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467577471b9c4e3ea7d633f414649bed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44907 } }
I20260924 03:27:03.168648 31790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 467577471b9c4e3ea7d633f414649bed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:27:03.168776 31790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 467577471b9c4e3ea7d633f414649bed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:27:03.168929 31790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 467577471b9c4e3ea7d633f414649bed [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:27:03.169922 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:27:03.171429 31790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 467577471b9c4e3ea7d633f414649bed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467577471b9c4e3ea7d633f414649bed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44907 } }
I20260924 03:27:03.171989 31790 leader_election.cc:302] T 00000000000000000000000000000000 P 467577471b9c4e3ea7d633f414649bed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 467577471b9c4e3ea7d633f414649bed; no voters:
I20260924 03:27:03.172636 31790 leader_election.cc:288] T 00000000000000000000000000000000 P 467577471b9c4e3ea7d633f414649bed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:27:03.172746 31796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 467577471b9c4e3ea7d633f414649bed [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:27:03.174177 31796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 467577471b9c4e3ea7d633f414649bed [term 1 LEADER]: Becoming Leader. State: Replica: 467577471b9c4e3ea7d633f414649bed, State: Running, Role: LEADER
I20260924 03:27:03.174830 31796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 467577471b9c4e3ea7d633f414649bed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467577471b9c4e3ea7d633f414649bed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44907 } }
I20260924 03:27:03.175038 31790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 467577471b9c4e3ea7d633f414649bed [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:27:03.178251 31801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467577471b9c4e3ea7d633f414649bed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 467577471b9c4e3ea7d633f414649bed. Latest consensus state: current_term: 1 leader_uuid: "467577471b9c4e3ea7d633f414649bed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467577471b9c4e3ea7d633f414649bed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44907 } } }
I20260924 03:27:03.178727 31801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467577471b9c4e3ea7d633f414649bed [sys.catalog]: This master's current role is: LEADER
I20260924 03:27:03.179353 31799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467577471b9c4e3ea7d633f414649bed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "467577471b9c4e3ea7d633f414649bed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467577471b9c4e3ea7d633f414649bed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44907 } } }
I20260924 03:27:03.179666 31799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467577471b9c4e3ea7d633f414649bed [sys.catalog]: This master's current role is: LEADER
I20260924 03:27:03.179663 31809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:27:03.192149 31809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:27:03.193425 31793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:03.175114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.212:56367 (local address 127.0.0.1:44907)
0924 03:27:03.175306 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:03.175312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:03.175689 (+ 377us) server_negotiation.cc:407] Connection header received
0924 03:27:03.175869 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:03.175874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:03.175944 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:03.176072 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:03.177866 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:03.179118 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:03.180976 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:03.181646 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:03.181757 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:03.185368 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:03.185375 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:03.185380 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:03.185425 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:03.188943 (+ 3518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:03.190031 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:03.190038 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:03.190042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:03.190226 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:03.191303 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:03.191309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:03.191312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:03.191737 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:03.192037 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:03.192760 (+ 723us) server_negotiation.cc:299] Negotiation successful
0924 03:27:03.192997 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 03:27:03.205227 31809 catalog_manager.cc:1347] Generated new cluster ID: 89467e2487d4423a96e68fe03b9d6a0b
I20260924 03:27:03.205463 31809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:27:03.226344 31809 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:27:03.227692 31809 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:27:03.237092 31809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 467577471b9c4e3ea7d633f414649bed: Generated new TSK 0
I20260924 03:27:03.238003 31809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:27:03.245028 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 31723
I20260924 03:27:03.247079 31785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:27:03.339607 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 31723
W20260924 03:27:03.353617 31708 connection.cc:535] client connection to 127.0.0.1:44907 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44907): Connection reset by peer (error 104)
2026-09-24T03:27:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1973 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:27:03 dist-test-slave-00qs krb5kdc[31822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:27:03 dist-test-slave-00qs krb5kdc[31822](info): set up 2 sockets
Sep 24 03:27:03 dist-test-slave-00qs krb5kdc[31822](info): commencing operation
krb5kdc: starting...
W20260924 03:27:04.388482 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:27:04 dist-test-slave-00qs krb5kdc[31822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:27:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:27:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:27:04.648175 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38757
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:27:04.985225 31835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:27:04.985594 31835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:27:04.985735 31835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:27:04.993295 31835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:27:04.993448 31835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:27:04.993526 31835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:27:04.993604 31835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:27:04.993669 31835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:27:05.005897 31835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38757
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:27:05.007331 31835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:27:05.008745 31835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:27:05.020956 31840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:05.020902 31841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:05.020882 31843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:27:05.021585 31835 server_base.cc:1034] running on GCE node
I20260924 03:27:05.022377 31835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:27:05.023902 31835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:27:05.025179 31835 hybrid_clock.cc:648] HybridClock initialized: now 1790220425025119 us; error 42 us; skew 500 ppm
I20260924 03:27:05.025566 31835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:27:05 dist-test-slave-00qs krb5kdc[31822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:27:05.033033 31835 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:27:05.034777 31835 webserver.cc:460] Webserver started at http://127.0.0.1:39931/ using document root <none> and password file <none>
I20260924 03:27:05.035480 31835 fs_manager.cc:359] Metadata directory not provided
I20260924 03:27:05.035588 31835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:27:05.035842 31835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:27:05.038902 31835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "92230990a29547ff9e7de0af3ed7cd84"
format_stamp: "Formatted at 2026-09-24 03:27:05 on dist-test-slave-00qs"
server_key: "24892390d4a9080131dae7737affeda5"
server_key_iv: "af3d95aa7c2a6c8bab94fedc3607dcdc"
server_key_version: "encryptionkey@0"
I20260924 03:27:05.039664 31835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "92230990a29547ff9e7de0af3ed7cd84"
format_stamp: "Formatted at 2026-09-24 03:27:05 on dist-test-slave-00qs"
server_key: "24892390d4a9080131dae7737affeda5"
server_key_iv: "af3d95aa7c2a6c8bab94fedc3607dcdc"
server_key_version: "encryptionkey@0"
I20260924 03:27:05.045215 31835 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 03:27:05.049817 31850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:27:05.051385 31835 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260924 03:27:05.051577 31835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "92230990a29547ff9e7de0af3ed7cd84"
format_stamp: "Formatted at 2026-09-24 03:27:05 on dist-test-slave-00qs"
server_key: "24892390d4a9080131dae7737affeda5"
server_key_iv: "af3d95aa7c2a6c8bab94fedc3607dcdc"
server_key_version: "encryptionkey@0"
I20260924 03:27:05.051786 31835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:27:05.070863 31835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:27:05.075372 31835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:27:05.075651 31835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:27:05.094331 31835 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38757
I20260924 03:27:05.094362 31901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38757 every 8 connection(s)
I20260924 03:27:05.095728 31835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:27:05.099866 31902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:27:05.103863 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 31835
I20260924 03:27:05.104110 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:27:05.104797 10290 external_mini_cluster.cc:1428] Setting key 0ea309bafe83222b1bf0cd5950d5c78f
I20260924 03:27:05.113059 31902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92230990a29547ff9e7de0af3ed7cd84: Bootstrap starting.
I20260924 03:27:05.116567 31902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92230990a29547ff9e7de0af3ed7cd84: Neither blocks nor log segments found. Creating new log.
I20260924 03:27:05.117811 31902 log.cc:824] T 00000000000000000000000000000000 P 92230990a29547ff9e7de0af3ed7cd84: Log is configured to *not* fsync() on all Append() calls
I20260924 03:27:05.118845 31905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:05.106710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.212:44255 (local address 127.0.0.1:38757)
0924 03:27:05.107524 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:05.107536 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:05.107578 (+ 42us) server_negotiation.cc:407] Connection header received
0924 03:27:05.109389 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:05.109416 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:05.109896 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:05.110332 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:05.110927 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:05.112299 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:05.113690 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:05.114395 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:05.114603 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:05.116158 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:05.116171 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:05.116187 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:27:05.116830 (+ 643us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:05.116933 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:27:05.116974 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:05.117140 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:05.117432 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 03:27:05.117611 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":347,"threads_started":1}
I20260924 03:27:05.121109 31902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92230990a29547ff9e7de0af3ed7cd84: No bootstrap required, opened a new log
I20260924 03:27:05.123234 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:27:05.129206 31902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92230990a29547ff9e7de0af3ed7cd84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92230990a29547ff9e7de0af3ed7cd84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38757 } }
I20260924 03:27:05.129748 31902 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92230990a29547ff9e7de0af3ed7cd84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:27:05.129871 31902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92230990a29547ff9e7de0af3ed7cd84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92230990a29547ff9e7de0af3ed7cd84, State: Initialized, Role: FOLLOWER
I20260924 03:27:05.130597 31902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92230990a29547ff9e7de0af3ed7cd84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92230990a29547ff9e7de0af3ed7cd84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38757 } }
I20260924 03:27:05.130832 31902 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92230990a29547ff9e7de0af3ed7cd84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:27:05.130937 31902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92230990a29547ff9e7de0af3ed7cd84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:27:05.131094 31902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92230990a29547ff9e7de0af3ed7cd84 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:27:05.133786 31902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92230990a29547ff9e7de0af3ed7cd84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92230990a29547ff9e7de0af3ed7cd84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38757 } }
I20260924 03:27:05.134433 31902 leader_election.cc:302] T 00000000000000000000000000000000 P 92230990a29547ff9e7de0af3ed7cd84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92230990a29547ff9e7de0af3ed7cd84; no voters:
I20260924 03:27:05.134750 31905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:05.127754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.212:34651 (local address 127.0.0.1:38757)
0924 03:27:05.128015 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:05.128021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:05.128378 (+ 357us) server_negotiation.cc:407] Connection header received
0924 03:27:05.128538 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:05.128543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:05.128630 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:05.128749 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:05.129299 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:05.130403 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:05.132043 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:05.132746 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:05.132881 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:05.133999 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:05.134006 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:05.134010 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:27:05.134032 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:05.134078 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:27:05.134099 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:05.134212 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:05.134368 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 03:27:05.134432 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:27:05.135186 31902 leader_election.cc:288] T 00000000000000000000000000000000 P 92230990a29547ff9e7de0af3ed7cd84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:27:05.135294 31912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92230990a29547ff9e7de0af3ed7cd84 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:27:05.136437 31912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92230990a29547ff9e7de0af3ed7cd84 [term 1 LEADER]: Becoming Leader. State: Replica: 92230990a29547ff9e7de0af3ed7cd84, State: Running, Role: LEADER
I20260924 03:27:05.137008 31912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92230990a29547ff9e7de0af3ed7cd84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92230990a29547ff9e7de0af3ed7cd84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38757 } }
I20260924 03:27:05.137360 31902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92230990a29547ff9e7de0af3ed7cd84 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:27:05.139724 31913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92230990a29547ff9e7de0af3ed7cd84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92230990a29547ff9e7de0af3ed7cd84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92230990a29547ff9e7de0af3ed7cd84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38757 } } }
I20260924 03:27:05.140146 31913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92230990a29547ff9e7de0af3ed7cd84 [sys.catalog]: This master's current role is: LEADER
I20260924 03:27:05.139611 31914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92230990a29547ff9e7de0af3ed7cd84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92230990a29547ff9e7de0af3ed7cd84. Latest consensus state: current_term: 1 leader_uuid: "92230990a29547ff9e7de0af3ed7cd84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92230990a29547ff9e7de0af3ed7cd84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38757 } } }
I20260924 03:27:05.141479 31914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92230990a29547ff9e7de0af3ed7cd84 [sys.catalog]: This master's current role is: LEADER
I20260924 03:27:05.143061 31921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:27:05.148602 31921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:27:05.159852 31921 catalog_manager.cc:1347] Generated new cluster ID: 8384ee60054f4d1787b8049fcf51d984
I20260924 03:27:05.160081 31921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:27:05.172508 31921 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:27:05.173893 31921 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:27:05.182047 31921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92230990a29547ff9e7de0af3ed7cd84: Generated new TSK 0
I20260924 03:27:05.182994 31921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:27:05.192034 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 31835
W20260924 03:27:05.193372 31897 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.212:44255
W20260924 03:27:05.194182 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 31835: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:27:05.194324 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 31835
W20260924 03:27:05.208374 31820 connection.cc:535] client connection to 127.0.0.1:38757 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38757): Connection reset by peer (error 104)
2026-09-24T03:27:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1854 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:27:05 dist-test-slave-00qs krb5kdc[31933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:27:05 dist-test-slave-00qs krb5kdc[31933](info): set up 2 sockets
Sep 24 03:27:05 dist-test-slave-00qs krb5kdc[31933](info): commencing operation
krb5kdc: starting...
W20260924 03:27:06.240490 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:27:06 dist-test-slave-00qs krb5kdc[31933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:27:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:27:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:27:06.500660 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35919
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:27:06.838954 31946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:27:06.839268 31946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:27:06.839396 31946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:27:06.847544 31946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:27:06.847697 31946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:27:06.847779 31946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:27:06.847868 31946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:27:06.847935 31946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:27:06.858407 31946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35919
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:27:06.859761 31946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:27:06.861105 31946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:27:06.873294 31952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:06.873354 31954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:06.873265 31951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:27:06.874023 31946 server_base.cc:1034] running on GCE node
I20260924 03:27:06.874815 31946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:27:06.876310 31946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:27:06.877564 31946 hybrid_clock.cc:648] HybridClock initialized: now 1790220426877505 us; error 51 us; skew 500 ppm
I20260924 03:27:06.877907 31946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:27:06 dist-test-slave-00qs krb5kdc[31933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:27:06.881122 31946 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:27:06.882809 31946 webserver.cc:460] Webserver started at http://127.0.0.1:33419/ using document root <none> and password file <none>
I20260924 03:27:06.883632 31946 fs_manager.cc:359] Metadata directory not provided
I20260924 03:27:06.883785 31946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:27:06.884188 31946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:27:06.887332 31946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "52fdd53da03943cf822356387a97ed48"
format_stamp: "Formatted at 2026-09-24 03:27:06 on dist-test-slave-00qs"
server_key: "6b1a6a16995c619132776bdeebd40a61"
server_key_iv: "0da8a33ca6493a2b562f9800acf37138"
server_key_version: "encryptionkey@0"
I20260924 03:27:06.888185 31946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "52fdd53da03943cf822356387a97ed48"
format_stamp: "Formatted at 2026-09-24 03:27:06 on dist-test-slave-00qs"
server_key: "6b1a6a16995c619132776bdeebd40a61"
server_key_iv: "0da8a33ca6493a2b562f9800acf37138"
server_key_version: "encryptionkey@0"
I20260924 03:27:06.893791 31946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 03:27:06.897584 31961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:27:06.898955 31946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 03:27:06.899114 31946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "52fdd53da03943cf822356387a97ed48"
format_stamp: "Formatted at 2026-09-24 03:27:06 on dist-test-slave-00qs"
server_key: "6b1a6a16995c619132776bdeebd40a61"
server_key_iv: "0da8a33ca6493a2b562f9800acf37138"
server_key_version: "encryptionkey@0"
I20260924 03:27:06.899300 31946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:27:06.905400 31946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:27:06.909808 31946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:27:06.910125 31946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:27:06.928876 31946 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35919
I20260924 03:27:06.928934 32012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35919 every 8 connection(s)
I20260924 03:27:06.930351 31946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:27:06.934170 32013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:27:06.936407 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 31946
I20260924 03:27:06.936599 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:27:06.937027 10290 external_mini_cluster.cc:1428] Setting key 4130403cb3764bbb185d41f4c1fe204b
I20260924 03:27:06.945106 32016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:06.938789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.212:54739 (local address 127.0.0.1:35919)
0924 03:27:06.939573 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:06.939585 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:06.939630 (+ 45us) server_negotiation.cc:407] Connection header received
0924 03:27:06.941507 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:06.941533 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:06.941902 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:06.942257 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:06.943376 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:06.943384 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:06.943396 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:27:06.943413 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:06.943494 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:27:06.943523 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:06.943725 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:06.943829 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 03:27:06.944099 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":340,"threads_started":1}
I20260924 03:27:06.946659 32013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52fdd53da03943cf822356387a97ed48: Bootstrap starting.
I20260924 03:27:06.949235 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:27:06.950026 32013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52fdd53da03943cf822356387a97ed48: Neither blocks nor log segments found. Creating new log.
I20260924 03:27:06.951073 32013 log.cc:824] T 00000000000000000000000000000000 P 52fdd53da03943cf822356387a97ed48: Log is configured to *not* fsync() on all Append() calls
I20260924 03:27:06.954721 32013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52fdd53da03943cf822356387a97ed48: No bootstrap required, opened a new log
I20260924 03:27:06.956272 32016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:06.953690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.212:33913 (local address 127.0.0.1:35919)
0924 03:27:06.953849 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:06.953855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:06.954238 (+ 383us) server_negotiation.cc:407] Connection header received
0924 03:27:06.954401 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:06.954462 (+ 61us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:06.954517 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:06.954604 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:06.955610 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:06.955614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:06.955617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:27:06.955633 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:06.955677 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:27:06.955693 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:06.955794 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:06.955909 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 03:27:06.956005 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 03:27:06.963384 32013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52fdd53da03943cf822356387a97ed48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52fdd53da03943cf822356387a97ed48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35919 } }
I20260924 03:27:06.963825 32013 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52fdd53da03943cf822356387a97ed48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:27:06.963927 32013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52fdd53da03943cf822356387a97ed48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52fdd53da03943cf822356387a97ed48, State: Initialized, Role: FOLLOWER
I20260924 03:27:06.964627 32013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52fdd53da03943cf822356387a97ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52fdd53da03943cf822356387a97ed48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35919 } }
I20260924 03:27:06.964918 32013 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52fdd53da03943cf822356387a97ed48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:27:06.965085 32013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52fdd53da03943cf822356387a97ed48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:27:06.965251 32013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52fdd53da03943cf822356387a97ed48 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:27:06.967555 32013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52fdd53da03943cf822356387a97ed48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52fdd53da03943cf822356387a97ed48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35919 } }
I20260924 03:27:06.968106 32013 leader_election.cc:302] T 00000000000000000000000000000000 P 52fdd53da03943cf822356387a97ed48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52fdd53da03943cf822356387a97ed48; no voters:
I20260924 03:27:06.968664 32013 leader_election.cc:288] T 00000000000000000000000000000000 P 52fdd53da03943cf822356387a97ed48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:27:06.968818 32023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52fdd53da03943cf822356387a97ed48 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:27:06.969471 32023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52fdd53da03943cf822356387a97ed48 [term 1 LEADER]: Becoming Leader. State: Replica: 52fdd53da03943cf822356387a97ed48, State: Running, Role: LEADER
I20260924 03:27:06.970007 32023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52fdd53da03943cf822356387a97ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52fdd53da03943cf822356387a97ed48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35919 } }
I20260924 03:27:06.970857 32013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52fdd53da03943cf822356387a97ed48 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:27:06.973827 32025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52fdd53da03943cf822356387a97ed48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52fdd53da03943cf822356387a97ed48. Latest consensus state: current_term: 1 leader_uuid: "52fdd53da03943cf822356387a97ed48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52fdd53da03943cf822356387a97ed48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35919 } } }
I20260924 03:27:06.973583 32024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52fdd53da03943cf822356387a97ed48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52fdd53da03943cf822356387a97ed48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52fdd53da03943cf822356387a97ed48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35919 } } }
I20260924 03:27:06.974411 32025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52fdd53da03943cf822356387a97ed48 [sys.catalog]: This master's current role is: LEADER
I20260924 03:27:06.974469 32024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52fdd53da03943cf822356387a97ed48 [sys.catalog]: This master's current role is: LEADER
I20260924 03:27:06.977201 32032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:27:06.981967 32032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:27:06.990782 32032 catalog_manager.cc:1347] Generated new cluster ID: ec23574b969e4880939f02a208a3fa74
I20260924 03:27:06.990912 32032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:27:07.006371 32032 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:27:07.006827 32032 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:27:07.017407 32032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52fdd53da03943cf822356387a97ed48: Generated new TSK 0
I20260924 03:27:07.018302 32032 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:27:07.023402 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 31946
W20260924 03:27:07.024606 32008 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.212:54739
W20260924 03:27:07.025424 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 31946: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:27:07.025560 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 31946
2026-09-24T03:27:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1831 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:27:07 dist-test-slave-00qs krb5kdc[32044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:27:07 dist-test-slave-00qs krb5kdc[32044](info): set up 2 sockets
Sep 24 03:27:07 dist-test-slave-00qs krb5kdc[32044](info): commencing operation
krb5kdc: starting...
W20260924 03:27:08.068533 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:27:08 dist-test-slave-00qs krb5kdc[32044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:27:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:27:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:27:08.328064 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:41195
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:40853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:41195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:27:08.665494 32057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:27:08.665845 32057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:27:08.665980 32057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:27:08.673542 32057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:27:08.673666 32057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:27:08.673725 32057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:27:08.673807 32057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:27:08.673864 32057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:27:08.683969 32057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:40853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:41195
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:41195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:27:08.685325 32057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:27:08.686663 32057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:27:08.699111 32063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:08.699226 32065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:08.699806 32062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:27:08.700060 32057 server_base.cc:1034] running on GCE node
I20260924 03:27:08.700979 32057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:27:08.702704 32057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:27:08.704002 32057 hybrid_clock.cc:648] HybridClock initialized: now 1790220428703932 us; error 42 us; skew 500 ppm
I20260924 03:27:08.704476 32057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:27:08 dist-test-slave-00qs krb5kdc[32044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220428, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:27:08.708043 32057 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:27:08.709596 32057 webserver.cc:460] Webserver started at http://127.10.12.190:42345/ using document root <none> and password file <none>
I20260924 03:27:08.710295 32057 fs_manager.cc:359] Metadata directory not provided
I20260924 03:27:08.710443 32057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:27:08.710731 32057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:27:08.713907 32057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "3c13839d3862479e977b8d8a0ce864c3"
format_stamp: "Formatted at 2026-09-24 03:27:08 on dist-test-slave-00qs"
server_key: "f2918f17f5643e856ce73ee9fda3823a"
server_key_iv: "359fcd6ec83cd435bddede7d50714cec"
server_key_version: "encryptionkey@0"
I20260924 03:27:08.714905 32057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "3c13839d3862479e977b8d8a0ce864c3"
format_stamp: "Formatted at 2026-09-24 03:27:08 on dist-test-slave-00qs"
server_key: "f2918f17f5643e856ce73ee9fda3823a"
server_key_iv: "359fcd6ec83cd435bddede7d50714cec"
server_key_version: "encryptionkey@0"
I20260924 03:27:08.720670 32057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 03:27:08.724568 32072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:27:08.726047 32057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:27:08.726238 32057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "3c13839d3862479e977b8d8a0ce864c3"
format_stamp: "Formatted at 2026-09-24 03:27:08 on dist-test-slave-00qs"
server_key: "f2918f17f5643e856ce73ee9fda3823a"
server_key_iv: "359fcd6ec83cd435bddede7d50714cec"
server_key_version: "encryptionkey@0"
I20260924 03:27:08.726557 32057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:27:08.760896 32057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:27:08.765654 32057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:27:08.765956 32057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:27:08.784638 32057 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:41195
I20260924 03:27:08.784631 32123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:41195 every 8 connection(s)
I20260924 03:27:08.786072 32057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:27:08.789952 32124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:27:08.794342 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 32057
I20260924 03:27:08.794540 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:27:08.795069 10290 external_mini_cluster.cc:1428] Setting key d8bba53ddf4e14af46cd14c3d789a810
I20260924 03:27:08.801879 32124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c13839d3862479e977b8d8a0ce864c3: Bootstrap starting.
I20260924 03:27:08.805637 32124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c13839d3862479e977b8d8a0ce864c3: Neither blocks nor log segments found. Creating new log.
I20260924 03:27:08.806850 32124 log.cc:824] T 00000000000000000000000000000000 P 3c13839d3862479e977b8d8a0ce864c3: Log is configured to *not* fsync() on all Append() calls
I20260924 03:27:08.810359 32124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c13839d3862479e977b8d8a0ce864c3: No bootstrap required, opened a new log
Sep 24 03:27:08 dist-test-slave-00qs krb5kdc[32044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:27:08.818764 32124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c13839d3862479e977b8d8a0ce864c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c13839d3862479e977b8d8a0ce864c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 41195 } }
I20260924 03:27:08.819204 32124 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c13839d3862479e977b8d8a0ce864c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:27:08.819309 32124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c13839d3862479e977b8d8a0ce864c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c13839d3862479e977b8d8a0ce864c3, State: Initialized, Role: FOLLOWER
I20260924 03:27:08.820000 32127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:08.796980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46389 (local address 127.10.12.190:41195)
0924 03:27:08.797792 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:08.797804 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:08.797849 (+ 45us) server_negotiation.cc:407] Connection header received
0924 03:27:08.800304 (+ 2455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:08.800331 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:08.800785 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:08.801124 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:08.802893 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:08.804329 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:08.806110 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:08.806913 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:08.807090 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:08.811870 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:08.811882 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:08.811901 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:08.811946 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:08.815338 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:08.816335 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:08.816346 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:08.816353 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:08.816554 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:08.817025 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:08.817029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:08.817031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:08.817651 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:08.817967 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:08.818485 (+ 518us) server_negotiation.cc:299] Negotiation successful
0924 03:27:08.818814 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":386,"threads_started":1}
I20260924 03:27:08.819984 32124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c13839d3862479e977b8d8a0ce864c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c13839d3862479e977b8d8a0ce864c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 41195 } }
I20260924 03:27:08.820286 32124 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c13839d3862479e977b8d8a0ce864c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:27:08.820425 32124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c13839d3862479e977b8d8a0ce864c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:27:08.820590 32124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c13839d3862479e977b8d8a0ce864c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:27:08.823207 32124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c13839d3862479e977b8d8a0ce864c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c13839d3862479e977b8d8a0ce864c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 41195 } }
I20260924 03:27:08.823704 32124 leader_election.cc:302] T 00000000000000000000000000000000 P 3c13839d3862479e977b8d8a0ce864c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c13839d3862479e977b8d8a0ce864c3; no voters:
I20260924 03:27:08.824048 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:27:08.824394 32124 leader_election.cc:288] T 00000000000000000000000000000000 P 3c13839d3862479e977b8d8a0ce864c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:27:08.824510 32129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c13839d3862479e977b8d8a0ce864c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:27:08.826766 32124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c13839d3862479e977b8d8a0ce864c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:27:08.828588 32129 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c13839d3862479e977b8d8a0ce864c3 [term 1 LEADER]: Becoming Leader. State: Replica: 3c13839d3862479e977b8d8a0ce864c3, State: Running, Role: LEADER
I20260924 03:27:08.829377 32129 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c13839d3862479e977b8d8a0ce864c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c13839d3862479e977b8d8a0ce864c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 41195 } }
I20260924 03:27:08.839344 32131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c13839d3862479e977b8d8a0ce864c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c13839d3862479e977b8d8a0ce864c3. Latest consensus state: current_term: 1 leader_uuid: "3c13839d3862479e977b8d8a0ce864c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c13839d3862479e977b8d8a0ce864c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 41195 } } }
I20260924 03:27:08.840175 32131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c13839d3862479e977b8d8a0ce864c3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:27:08.841822 32130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c13839d3862479e977b8d8a0ce864c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c13839d3862479e977b8d8a0ce864c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c13839d3862479e977b8d8a0ce864c3" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 41195 } } }
I20260924 03:27:08.842149 32130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c13839d3862479e977b8d8a0ce864c3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:27:08.845310 32127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:08.829416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53741 (local address 127.10.12.190:41195)
0924 03:27:08.829651 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:08.829657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:08.829676 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:27:08.829734 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:08.829739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:08.829807 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:08.829891 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:08.831885 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:08.833150 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:08.834567 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:08.835069 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:08.835180 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:08.837751 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:08.837759 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:08.837764 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:08.837813 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:08.841553 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:08.842795 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:08.842805 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:08.842809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:08.843030 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:08.843584 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:08.843589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:08.843592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:08.844142 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:08.844426 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:08.844753 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 03:27:08.844962 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
W20260924 03:27:08.848694 32150 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c13839d3862479e977b8d8a0ce864c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:27:08.848845 32150 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:27:08.849181 32147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:27:08.850914 32147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:27:08.860355 32147 catalog_manager.cc:1347] Generated new cluster ID: 20f3c29b0beb4bdcad13b493c225565e
I20260924 03:27:08.860553 32147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:27:08.869191 32147 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:27:08.870471 32147 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:27:08.880556 32147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c13839d3862479e977b8d8a0ce864c3: Generated new TSK 0
I20260924 03:27:08.881395 32147 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:27:08.890612 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 32057
I20260924 03:27:08.892136 32119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:27:08.985460 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 32057
W20260924 03:27:08.999091 32042 connection.cc:535] client connection to 127.10.12.190:41195 recv error: Network error: failed to read from TLS socket (remote: 127.10.12.190:41195): Connection reset by peer (error 104)
2026-09-24T03:27:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1959 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:27:09 dist-test-slave-00qs krb5kdc[32156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:27:09 dist-test-slave-00qs krb5kdc[32156](info): set up 2 sockets
Sep 24 03:27:09 dist-test-slave-00qs krb5kdc[32156](info): commencing operation
krb5kdc: starting...
W20260924 03:27:10.020463 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:27:10 dist-test-slave-00qs krb5kdc[32156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:27:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:27:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:27:10.286346 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:38093
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:37859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:38093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:27:10.623587 32169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:27:10.623917 32169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:27:10.624068 32169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:27:10.631556 32169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:27:10.631698 32169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:27:10.631778 32169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:27:10.631862 32169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:27:10.631922 32169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:27:10.642159 32169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:37859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:38093
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:38093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:27:10.643508 32169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:27:10.644919 32169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:27:10.656610 32174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:10.656672 32177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:10.656708 32175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:27:10.657482 32169 server_base.cc:1034] running on GCE node
I20260924 03:27:10.658293 32169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:27:10.659811 32169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:27:10.661098 32169 hybrid_clock.cc:648] HybridClock initialized: now 1790220430661064 us; error 64 us; skew 500 ppm
I20260924 03:27:10.661489 32169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:27:10 dist-test-slave-00qs krb5kdc[32156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220430, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:27:10.668735 32169 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:27:10.670496 32169 webserver.cc:460] Webserver started at http://127.10.12.190:40497/ using document root <none> and password file <none>
I20260924 03:27:10.671231 32169 fs_manager.cc:359] Metadata directory not provided
I20260924 03:27:10.671360 32169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:27:10.671671 32169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:27:10.674907 32169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "d8a3f42886484181a0f6cfc2d73ffc63"
format_stamp: "Formatted at 2026-09-24 03:27:10 on dist-test-slave-00qs"
server_key: "1ed5709392fedc4989bebed0f23c40e9"
server_key_iv: "18a7b6bad55d49d21822c63d65d95754"
server_key_version: "encryptionkey@0"
I20260924 03:27:10.675688 32169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "d8a3f42886484181a0f6cfc2d73ffc63"
format_stamp: "Formatted at 2026-09-24 03:27:10 on dist-test-slave-00qs"
server_key: "1ed5709392fedc4989bebed0f23c40e9"
server_key_iv: "18a7b6bad55d49d21822c63d65d95754"
server_key_version: "encryptionkey@0"
I20260924 03:27:10.681161 32169 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 03:27:10.684967 32184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:27:10.686422 32169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:27:10.686668 32169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "d8a3f42886484181a0f6cfc2d73ffc63"
format_stamp: "Formatted at 2026-09-24 03:27:10 on dist-test-slave-00qs"
server_key: "1ed5709392fedc4989bebed0f23c40e9"
server_key_iv: "18a7b6bad55d49d21822c63d65d95754"
server_key_version: "encryptionkey@0"
I20260924 03:27:10.686874 32169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:27:10.705482 32169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:27:10.710202 32169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:27:10.710518 32169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:27:10.728623 32169 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:38093
I20260924 03:27:10.728623 32235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:38093 every 8 connection(s)
I20260924 03:27:10.730266 32169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:27:10.732197 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 32169
I20260924 03:27:10.732357 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:27:10.732820 10290 external_mini_cluster.cc:1428] Setting key 34ff5ab9b8d4f663a39494fad8166ac3
I20260924 03:27:10.734926 32236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:27:10.747933 32239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:10.734728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39633 (local address 127.10.12.190:38093)
0924 03:27:10.735568 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:10.735579 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:10.735621 (+ 42us) server_negotiation.cc:407] Connection header received
0924 03:27:10.737336 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:10.737371 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:10.737819 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:10.738210 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:10.738921 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:10.740363 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:10.742371 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:10.743180 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:10.743329 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:10.745039 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:10.745051 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:10.745079 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:27:10.746023 (+ 944us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:10.746120 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:27:10.746156 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:10.746325 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:10.746662 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 03:27:10.746874 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":377,"threads_started":1}
I20260924 03:27:10.748332 32236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8a3f42886484181a0f6cfc2d73ffc63: Bootstrap starting.
I20260924 03:27:10.751679 32236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8a3f42886484181a0f6cfc2d73ffc63: Neither blocks nor log segments found. Creating new log.
I20260924 03:27:10.752346 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:27:10.752887 32236 log.cc:824] T 00000000000000000000000000000000 P d8a3f42886484181a0f6cfc2d73ffc63: Log is configured to *not* fsync() on all Append() calls
I20260924 03:27:10.756094 32236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8a3f42886484181a0f6cfc2d73ffc63: No bootstrap required, opened a new log
I20260924 03:27:10.763507 32239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:10.757004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32841 (local address 127.10.12.190:38093)
0924 03:27:10.757179 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:10.757185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:10.757591 (+ 406us) server_negotiation.cc:407] Connection header received
0924 03:27:10.757768 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:10.757772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:10.757839 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:10.757930 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:10.758440 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:10.759243 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:10.760608 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:10.761079 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:10.761204 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:10.762792 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:10.762797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:10.762801 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:27:10.762823 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:10.762870 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:27:10.762886 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:10.763018 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:10.763183 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 03:27:10.763243 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 03:27:10.764521 32236 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8a3f42886484181a0f6cfc2d73ffc63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a3f42886484181a0f6cfc2d73ffc63" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 38093 } }
I20260924 03:27:10.764966 32236 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8a3f42886484181a0f6cfc2d73ffc63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:27:10.765089 32236 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8a3f42886484181a0f6cfc2d73ffc63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8a3f42886484181a0f6cfc2d73ffc63, State: Initialized, Role: FOLLOWER
I20260924 03:27:10.765652 32236 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8a3f42886484181a0f6cfc2d73ffc63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a3f42886484181a0f6cfc2d73ffc63" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 38093 } }
I20260924 03:27:10.765861 32236 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8a3f42886484181a0f6cfc2d73ffc63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:27:10.765964 32236 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8a3f42886484181a0f6cfc2d73ffc63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:27:10.766088 32236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8a3f42886484181a0f6cfc2d73ffc63 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:27:10.768573 32236 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8a3f42886484181a0f6cfc2d73ffc63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a3f42886484181a0f6cfc2d73ffc63" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 38093 } }
I20260924 03:27:10.769129 32236 leader_election.cc:302] T 00000000000000000000000000000000 P d8a3f42886484181a0f6cfc2d73ffc63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8a3f42886484181a0f6cfc2d73ffc63; no voters:
I20260924 03:27:10.769716 32236 leader_election.cc:288] T 00000000000000000000000000000000 P d8a3f42886484181a0f6cfc2d73ffc63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:27:10.769827 32246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8a3f42886484181a0f6cfc2d73ffc63 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:27:10.770565 32246 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8a3f42886484181a0f6cfc2d73ffc63 [term 1 LEADER]: Becoming Leader. State: Replica: d8a3f42886484181a0f6cfc2d73ffc63, State: Running, Role: LEADER
I20260924 03:27:10.771157 32246 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8a3f42886484181a0f6cfc2d73ffc63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a3f42886484181a0f6cfc2d73ffc63" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 38093 } }
I20260924 03:27:10.771803 32236 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8a3f42886484181a0f6cfc2d73ffc63 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:27:10.773892 32247 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8a3f42886484181a0f6cfc2d73ffc63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8a3f42886484181a0f6cfc2d73ffc63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a3f42886484181a0f6cfc2d73ffc63" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 38093 } } }
I20260924 03:27:10.773867 32248 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8a3f42886484181a0f6cfc2d73ffc63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8a3f42886484181a0f6cfc2d73ffc63. Latest consensus state: current_term: 1 leader_uuid: "d8a3f42886484181a0f6cfc2d73ffc63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a3f42886484181a0f6cfc2d73ffc63" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 38093 } } }
I20260924 03:27:10.774317 32247 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8a3f42886484181a0f6cfc2d73ffc63 [sys.catalog]: This master's current role is: LEADER
I20260924 03:27:10.774317 32248 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8a3f42886484181a0f6cfc2d73ffc63 [sys.catalog]: This master's current role is: LEADER
I20260924 03:27:10.775161 32252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:27:10.780915 32252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:27:10.793956 32252 catalog_manager.cc:1347] Generated new cluster ID: b67b625599924afebd1c98e001c46ef4
I20260924 03:27:10.794106 32252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:27:10.811434 32252 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:27:10.812893 32252 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:27:10.822118 32252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8a3f42886484181a0f6cfc2d73ffc63: Generated new TSK 0
I20260924 03:27:10.823293 32252 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:27:10.839442 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 32169
W20260924 03:27:10.840736 32231 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39633
W20260924 03:27:10.841686 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 32169: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:27:10.841867 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 32169
W20260924 03:27:10.856218 32154 connection.cc:535] client connection to 127.10.12.190:38093 recv error: Network error: failed to read from TLS socket (remote: 127.10.12.190:38093): Connection reset by peer (error 104)
2026-09-24T03:27:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1857 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:27:10 dist-test-slave-00qs krb5kdc[32267](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:27:10 dist-test-slave-00qs krb5kdc[32267](info): set up 2 sockets
Sep 24 03:27:10 dist-test-slave-00qs krb5kdc[32267](info): commencing operation
krb5kdc: starting...
W20260924 03:27:11.872500 10290 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:27:12 dist-test-slave-00qs krb5kdc[32267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:27:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:27:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:27:12.139596 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:45785
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:43217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:45785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:27:12.478479 32280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:27:12.478802 32280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:27:12.478946 32280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:27:12.486384 32280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:27:12.486531 32280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:27:12.486611 32280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:27:12.486694 32280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:27:12.486755 32280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:27:12.496974 32280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:43217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:45785
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:45785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:27:12.498332 32280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:27:12.499682 32280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:27:12.511933 32285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:12.511893 32286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:12.511859 32288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:27:12.512797 32280 server_base.cc:1034] running on GCE node
I20260924 03:27:12.513602 32280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:27:12.515237 32280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:27:12.516479 32280 hybrid_clock.cc:648] HybridClock initialized: now 1790220432516422 us; error 46 us; skew 500 ppm
I20260924 03:27:12.516853 32280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:27:12 dist-test-slave-00qs krb5kdc[32267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220432, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:27:12.520351 32280 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:27:12.521978 32280 webserver.cc:460] Webserver started at http://127.10.12.190:36847/ using document root <none> and password file <none>
I20260924 03:27:12.522682 32280 fs_manager.cc:359] Metadata directory not provided
I20260924 03:27:12.522796 32280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:27:12.523103 32280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:27:12.526229 32280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "e49d9bc5982f48c3a12ea2ac9c944aea"
format_stamp: "Formatted at 2026-09-24 03:27:12 on dist-test-slave-00qs"
server_key: "1606c0ed0dd5d4ee279ba1d0d0dee294"
server_key_iv: "b77b1646a84d4eb5ac07d98f1887f5b8"
server_key_version: "encryptionkey@0"
I20260924 03:27:12.527040 32280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "e49d9bc5982f48c3a12ea2ac9c944aea"
format_stamp: "Formatted at 2026-09-24 03:27:12 on dist-test-slave-00qs"
server_key: "1606c0ed0dd5d4ee279ba1d0d0dee294"
server_key_iv: "b77b1646a84d4eb5ac07d98f1887f5b8"
server_key_version: "encryptionkey@0"
I20260924 03:27:12.532464 32280 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:27:12.536633 32295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:27:12.538096 32280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:27:12.538280 32280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "e49d9bc5982f48c3a12ea2ac9c944aea"
format_stamp: "Formatted at 2026-09-24 03:27:12 on dist-test-slave-00qs"
server_key: "1606c0ed0dd5d4ee279ba1d0d0dee294"
server_key_iv: "b77b1646a84d4eb5ac07d98f1887f5b8"
server_key_version: "encryptionkey@0"
I20260924 03:27:12.538504 32280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:27:12.544747 32280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:27:12.549394 32280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:27:12.549691 32280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:27:12.569293 32280 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:45785
I20260924 03:27:12.569322 32346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:45785 every 8 connection(s)
I20260924 03:27:12.570773 32280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:27:12.574643 32347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:27:12.574993 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 32280
I20260924 03:27:12.575145 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:27:12.575575 10290 external_mini_cluster.cc:1428] Setting key 3c2ceac727fffec40db18bfafaf4c8be
I20260924 03:27:12.584079 32350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:12.577359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38885 (local address 127.10.12.190:45785)
0924 03:27:12.578155 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:12.578167 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:12.578210 (+ 43us) server_negotiation.cc:407] Connection header received
0924 03:27:12.580528 (+ 2318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:12.580555 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:12.580947 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:12.581309 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:12.582492 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:12.582498 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:12.582512 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:27:12.582528 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:12.582590 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:27:12.582611 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:12.582783 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:12.582888 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 03:27:12.583074 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":340,"threads_started":1}
I20260924 03:27:12.588189 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:27:12.588727 32347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e49d9bc5982f48c3a12ea2ac9c944aea: Bootstrap starting.
I20260924 03:27:12.591820 32347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e49d9bc5982f48c3a12ea2ac9c944aea: Neither blocks nor log segments found. Creating new log.
I20260924 03:27:12.593116 32347 log.cc:824] T 00000000000000000000000000000000 P e49d9bc5982f48c3a12ea2ac9c944aea: Log is configured to *not* fsync() on all Append() calls
I20260924 03:27:12.595140 32350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:12.592597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46511 (local address 127.10.12.190:45785)
0924 03:27:12.592812 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:12.592820 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:12.593242 (+ 422us) server_negotiation.cc:407] Connection header received
0924 03:27:12.593467 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:12.593471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:12.593525 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:12.593609 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:12.594543 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:12.594547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:12.594550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:27:12.594564 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:12.594604 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:27:12.594620 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:12.594707 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:12.594815 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 03:27:12.594862 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:27:12.596419 32347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e49d9bc5982f48c3a12ea2ac9c944aea: No bootstrap required, opened a new log
I20260924 03:27:12.604467 32347 raft_consensus.cc:348] T 00000000000000000000000000000000 P e49d9bc5982f48c3a12ea2ac9c944aea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49d9bc5982f48c3a12ea2ac9c944aea" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45785 } }
I20260924 03:27:12.604887 32347 raft_consensus.cc:374] T 00000000000000000000000000000000 P e49d9bc5982f48c3a12ea2ac9c944aea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:27:12.604990 32347 raft_consensus.cc:729] T 00000000000000000000000000000000 P e49d9bc5982f48c3a12ea2ac9c944aea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49d9bc5982f48c3a12ea2ac9c944aea, State: Initialized, Role: FOLLOWER
I20260924 03:27:12.605619 32347 consensus_queue.cc:260] T 00000000000000000000000000000000 P e49d9bc5982f48c3a12ea2ac9c944aea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49d9bc5982f48c3a12ea2ac9c944aea" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45785 } }
I20260924 03:27:12.605856 32347 raft_consensus.cc:388] T 00000000000000000000000000000000 P e49d9bc5982f48c3a12ea2ac9c944aea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:27:12.605983 32347 raft_consensus.cc:482] T 00000000000000000000000000000000 P e49d9bc5982f48c3a12ea2ac9c944aea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:27:12.606144 32347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e49d9bc5982f48c3a12ea2ac9c944aea [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:27:12.609014 32347 raft_consensus.cc:504] T 00000000000000000000000000000000 P e49d9bc5982f48c3a12ea2ac9c944aea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49d9bc5982f48c3a12ea2ac9c944aea" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45785 } }
I20260924 03:27:12.609544 32347 leader_election.cc:302] T 00000000000000000000000000000000 P e49d9bc5982f48c3a12ea2ac9c944aea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e49d9bc5982f48c3a12ea2ac9c944aea; no voters:
I20260924 03:27:12.610131 32347 leader_election.cc:288] T 00000000000000000000000000000000 P e49d9bc5982f48c3a12ea2ac9c944aea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:27:12.610257 32357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e49d9bc5982f48c3a12ea2ac9c944aea [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:27:12.610903 32357 raft_consensus.cc:686] T 00000000000000000000000000000000 P e49d9bc5982f48c3a12ea2ac9c944aea [term 1 LEADER]: Becoming Leader. State: Replica: e49d9bc5982f48c3a12ea2ac9c944aea, State: Running, Role: LEADER
I20260924 03:27:12.611495 32357 consensus_queue.cc:237] T 00000000000000000000000000000000 P e49d9bc5982f48c3a12ea2ac9c944aea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49d9bc5982f48c3a12ea2ac9c944aea" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45785 } }
I20260924 03:27:12.612242 32347 sys_catalog.cc:564] T 00000000000000000000000000000000 P e49d9bc5982f48c3a12ea2ac9c944aea [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:27:12.614123 32358 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49d9bc5982f48c3a12ea2ac9c944aea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e49d9bc5982f48c3a12ea2ac9c944aea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49d9bc5982f48c3a12ea2ac9c944aea" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45785 } } }
I20260924 03:27:12.614535 32358 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49d9bc5982f48c3a12ea2ac9c944aea [sys.catalog]: This master's current role is: LEADER
I20260924 03:27:12.614615 32359 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49d9bc5982f48c3a12ea2ac9c944aea [sys.catalog]: SysCatalogTable state changed. Reason: New leader e49d9bc5982f48c3a12ea2ac9c944aea. Latest consensus state: current_term: 1 leader_uuid: "e49d9bc5982f48c3a12ea2ac9c944aea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49d9bc5982f48c3a12ea2ac9c944aea" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45785 } } }
I20260924 03:27:12.614881 32359 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49d9bc5982f48c3a12ea2ac9c944aea [sys.catalog]: This master's current role is: LEADER
I20260924 03:27:12.615862 32364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:27:12.620709 32364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:27:12.633368 32364 catalog_manager.cc:1347] Generated new cluster ID: 507bea2331b54217adb1fa18186a94ab
I20260924 03:27:12.633502 32364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:27:12.662029 32364 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:27:12.662463 32364 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:27:12.677307 32364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e49d9bc5982f48c3a12ea2ac9c944aea: Generated new TSK 0
I20260924 03:27:12.678196 32364 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:27:12.689729 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 32280
W20260924 03:27:12.690969 32342 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38885
W20260924 03:27:12.691798 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 32280: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:27:12.691978 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 32280
2026-09-24T03:27:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1848 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:27:12 dist-test-slave-00qs krb5kdc[32378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:27:12 dist-test-slave-00qs krb5kdc[32378](info): set up 2 sockets
Sep 24 03:27:12 dist-test-slave-00qs krb5kdc[32378](info): commencing operation
krb5kdc: starting...
W20260924 03:27:13.736510 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.010s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:27:13 dist-test-slave-00qs krb5kdc[32378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:27:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:27:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:27:13.996632 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:45623
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:40113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:45623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:27:14.333869 32391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:27:14.334173 32391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:27:14.334297 32391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:27:14.341900 32391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:27:14.342028 32391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:27:14.342100 32391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:27:14.342181 32391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:27:14.342242 32391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:27:14.352327 32391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:40113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:45623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:45623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:27:14.353665 32391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:27:14.354976 32391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:27:14.367218 32396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:14.367148 32399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:14.367260 32397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:27:14.367646 32391 server_base.cc:1034] running on GCE node
I20260924 03:27:14.369041 32391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:27:14.370662 32391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:27:14.371908 32391 hybrid_clock.cc:648] HybridClock initialized: now 1790220434371849 us; error 57 us; skew 500 ppm
I20260924 03:27:14.372341 32391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:27:14 dist-test-slave-00qs krb5kdc[32378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220434, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:27:14.376425 32391 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:27:14.378489 32391 webserver.cc:460] Webserver started at http://127.10.12.190:44115/ using document root <none> and password file <none>
I20260924 03:27:14.379228 32391 fs_manager.cc:359] Metadata directory not provided
I20260924 03:27:14.379347 32391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:27:14.379671 32391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:27:14.383515 32391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "55f27225aa9f46f397148a6181c73458"
format_stamp: "Formatted at 2026-09-24 03:27:14 on dist-test-slave-00qs"
server_key: "897ec7e6015bf6b597d3e50d97c32128"
server_key_iv: "9adb3e429b21451e1b671150ae94dc0c"
server_key_version: "encryptionkey@0"
I20260924 03:27:14.384485 32391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "55f27225aa9f46f397148a6181c73458"
format_stamp: "Formatted at 2026-09-24 03:27:14 on dist-test-slave-00qs"
server_key: "897ec7e6015bf6b597d3e50d97c32128"
server_key_iv: "9adb3e429b21451e1b671150ae94dc0c"
server_key_version: "encryptionkey@0"
I20260924 03:27:14.390286 32391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 03:27:14.394624 32406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:27:14.396317 32391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:27:14.396559 32391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "55f27225aa9f46f397148a6181c73458"
format_stamp: "Formatted at 2026-09-24 03:27:14 on dist-test-slave-00qs"
server_key: "897ec7e6015bf6b597d3e50d97c32128"
server_key_iv: "9adb3e429b21451e1b671150ae94dc0c"
server_key_version: "encryptionkey@0"
I20260924 03:27:14.396757 32391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:27:14.412312 32391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:27:14.417068 32391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:27:14.417539 32391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:27:14.437206 32457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:45623 every 8 connection(s)
I20260924 03:27:14.437199 32391 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:45623
I20260924 03:27:14.438843 32391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:27:14.442637 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 32391
I20260924 03:27:14.442826 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:27:14.442982 32458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:27:14.443373 10290 external_mini_cluster.cc:1428] Setting key a354edcc2b71dc9fbdf9cf27bde90b02
I20260924 03:27:14.456382 32458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55f27225aa9f46f397148a6181c73458: Bootstrap starting.
Sep 24 03:27:14 dist-test-slave-00qs krb5kdc[32378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:27:14.460129 32458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55f27225aa9f46f397148a6181c73458: Neither blocks nor log segments found. Creating new log.
I20260924 03:27:14.461412 32458 log.cc:824] T 00000000000000000000000000000000 P 55f27225aa9f46f397148a6181c73458: Log is configured to *not* fsync() on all Append() calls
I20260924 03:27:14.464814 32458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55f27225aa9f46f397148a6181c73458: No bootstrap required, opened a new log
I20260924 03:27:14.468259 32461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:14.445307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36485 (local address 127.10.12.190:45623)
0924 03:27:14.446192 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:14.446206 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:14.446253 (+ 47us) server_negotiation.cc:407] Connection header received
0924 03:27:14.448007 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:14.448037 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:14.448496 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:14.449084 (+ 588us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:14.450646 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:14.451944 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:14.453274 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:14.454043 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:14.454237 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:14.459283 (+ 5046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:14.459298 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:14.459327 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:14.459380 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:14.462719 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:14.463705 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:14.463716 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:14.463724 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:14.463919 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:14.464517 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:14.464526 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:14.464529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:14.465542 (+ 1013us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:14.466016 (+ 474us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:14.466548 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 03:27:14.467063 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":396,"threads_started":1}
I20260924 03:27:14.472692 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:27:14.474459 32458 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55f27225aa9f46f397148a6181c73458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f27225aa9f46f397148a6181c73458" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45623 } }
I20260924 03:27:14.474946 32458 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55f27225aa9f46f397148a6181c73458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:27:14.475052 32458 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55f27225aa9f46f397148a6181c73458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55f27225aa9f46f397148a6181c73458, State: Initialized, Role: FOLLOWER
I20260924 03:27:14.475751 32458 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55f27225aa9f46f397148a6181c73458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f27225aa9f46f397148a6181c73458" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45623 } }
I20260924 03:27:14.476111 32458 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55f27225aa9f46f397148a6181c73458 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:27:14.476267 32458 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55f27225aa9f46f397148a6181c73458 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:27:14.476464 32458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55f27225aa9f46f397148a6181c73458 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:27:14.479640 32458 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55f27225aa9f46f397148a6181c73458 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f27225aa9f46f397148a6181c73458" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45623 } }
I20260924 03:27:14.480345 32458 leader_election.cc:302] T 00000000000000000000000000000000 P 55f27225aa9f46f397148a6181c73458 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55f27225aa9f46f397148a6181c73458; no voters:
I20260924 03:27:14.481119 32458 leader_election.cc:288] T 00000000000000000000000000000000 P 55f27225aa9f46f397148a6181c73458 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:27:14.481273 32468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55f27225aa9f46f397148a6181c73458 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:27:14.483371 32458 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55f27225aa9f46f397148a6181c73458 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:27:14.488631 32468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55f27225aa9f46f397148a6181c73458 [term 1 LEADER]: Becoming Leader. State: Replica: 55f27225aa9f46f397148a6181c73458, State: Running, Role: LEADER
I20260924 03:27:14.489393 32468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55f27225aa9f46f397148a6181c73458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f27225aa9f46f397148a6181c73458" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45623 } }
I20260924 03:27:14.492292 32470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55f27225aa9f46f397148a6181c73458 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55f27225aa9f46f397148a6181c73458. Latest consensus state: current_term: 1 leader_uuid: "55f27225aa9f46f397148a6181c73458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f27225aa9f46f397148a6181c73458" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45623 } } }
I20260924 03:27:14.492720 32470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55f27225aa9f46f397148a6181c73458 [sys.catalog]: This master's current role is: LEADER
I20260924 03:27:14.492717 32469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55f27225aa9f46f397148a6181c73458 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55f27225aa9f46f397148a6181c73458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f27225aa9f46f397148a6181c73458" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 45623 } } }
I20260924 03:27:14.493041 32469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55f27225aa9f46f397148a6181c73458 [sys.catalog]: This master's current role is: LEADER
I20260924 03:27:14.494550 32482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:27:14.495445 32461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:14.477941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59483 (local address 127.10.12.190:45623)
0924 03:27:14.478167 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:14.478175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:14.478597 (+ 422us) server_negotiation.cc:407] Connection header received
0924 03:27:14.478770 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:14.478773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:14.478839 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:14.478928 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:14.480894 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:14.482189 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:14.483901 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:14.484756 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:14.485646 (+ 890us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:14.488316 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:14.488324 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:14.488329 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:14.488376 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:14.491786 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:14.492603 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:14.492610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:14.492614 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:14.492798 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:14.493318 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:14.493323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:14.493326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:14.493829 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:14.494759 (+ 930us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:14.494929 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 03:27:14.495139 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 03:27:14.500066 32482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:27:14.508749 32482 catalog_manager.cc:1347] Generated new cluster ID: 5df928be6e8a4d438a5fdd98a1447c14
I20260924 03:27:14.508872 32482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:27:14.530872 32482 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:27:14.532924 32482 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:27:14.549142 32482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55f27225aa9f46f397148a6181c73458: Generated new TSK 0
I20260924 03:27:14.550280 32482 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:27:14.560184 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 32391
I20260924 03:27:14.561417 32453 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:27:14.654429 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 32391
2026-09-24T03:27:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1962 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:27:14 dist-test-slave-00qs krb5kdc[32490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:27:14 dist-test-slave-00qs krb5kdc[32490](info): set up 2 sockets
Sep 24 03:27:14 dist-test-slave-00qs krb5kdc[32490](info): commencing operation
krb5kdc: starting...
W20260924 03:27:15.688736 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:27:15 dist-test-slave-00qs krb5kdc[32490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:27:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:27:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:27:15.947618 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:36757
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:35543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:36757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:27:16.289795 32503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:27:16.290122 32503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:27:16.290246 32503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:27:16.297823 32503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:27:16.297951 32503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:27:16.297998 32503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:27:16.298079 32503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:27:16.298139 32503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:27:16.308881 32503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:35543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:36757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:36757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:27:16.310237 32503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:27:16.311606 32503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:27:16.323697 32511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:16.323675 32509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:16.324218 32508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:27:16.324494 32503 server_base.cc:1034] running on GCE node
I20260924 03:27:16.325281 32503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:27:16.326732 32503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:27:16.328033 32503 hybrid_clock.cc:648] HybridClock initialized: now 1790220436327906 us; error 39 us; skew 500 ppm
I20260924 03:27:16.328402 32503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:27:16 dist-test-slave-00qs krb5kdc[32490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220436, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:27:16.336383 32503 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:27:16.338099 32503 webserver.cc:460] Webserver started at http://127.10.12.190:41343/ using document root <none> and password file <none>
I20260924 03:27:16.338805 32503 fs_manager.cc:359] Metadata directory not provided
I20260924 03:27:16.338936 32503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:27:16.339241 32503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:27:16.342548 32503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "aecf840a944b4b159935a25676d1c54b"
format_stamp: "Formatted at 2026-09-24 03:27:16 on dist-test-slave-00qs"
server_key: "f55c2f42af628583751edfeff1277853"
server_key_iv: "885ac3f623d1ba3dadac192a9a092296"
server_key_version: "encryptionkey@0"
I20260924 03:27:16.343405 32503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "aecf840a944b4b159935a25676d1c54b"
format_stamp: "Formatted at 2026-09-24 03:27:16 on dist-test-slave-00qs"
server_key: "f55c2f42af628583751edfeff1277853"
server_key_iv: "885ac3f623d1ba3dadac192a9a092296"
server_key_version: "encryptionkey@0"
I20260924 03:27:16.349208 32503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 03:27:16.354012 32518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:27:16.356216 32503 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.002s
I20260924 03:27:16.356442 32503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "aecf840a944b4b159935a25676d1c54b"
format_stamp: "Formatted at 2026-09-24 03:27:16 on dist-test-slave-00qs"
server_key: "f55c2f42af628583751edfeff1277853"
server_key_iv: "885ac3f623d1ba3dadac192a9a092296"
server_key_version: "encryptionkey@0"
I20260924 03:27:16.356676 32503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:27:16.370857 32503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:27:16.375061 32503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:27:16.375339 32503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:27:16.394526 32503 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:36757
I20260924 03:27:16.394544 32569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:36757 every 8 connection(s)
I20260924 03:27:16.395915 32503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:27:16.399736 32570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:27:16.403141 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 32503
I20260924 03:27:16.403318 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:27:16.403764 10290 external_mini_cluster.cc:1428] Setting key df7605688548afa95f34f5c5db0d5279
I20260924 03:27:16.412335 32570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aecf840a944b4b159935a25676d1c54b: Bootstrap starting.
I20260924 03:27:16.415469 32570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aecf840a944b4b159935a25676d1c54b: Neither blocks nor log segments found. Creating new log.
I20260924 03:27:16.416731 32570 log.cc:824] T 00000000000000000000000000000000 P aecf840a944b4b159935a25676d1c54b: Log is configured to *not* fsync() on all Append() calls
I20260924 03:27:16.417409 32573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:16.405945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45061 (local address 127.10.12.190:36757)
0924 03:27:16.406728 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:16.406740 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:16.406783 (+ 43us) server_negotiation.cc:407] Connection header received
0924 03:27:16.408662 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:16.408688 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:16.409180 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:16.409588 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:16.410191 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:16.411511 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:16.412912 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:16.413589 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:16.413794 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:16.414901 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:16.414909 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:16.414925 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:27:16.415477 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:16.415557 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:27:16.415587 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:16.415709 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:16.416015 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 03:27:16.416299 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":356,"threads_started":1}
I20260924 03:27:16.419814 32570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aecf840a944b4b159935a25676d1c54b: No bootstrap required, opened a new log
I20260924 03:27:16.421599 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:27:16.428139 32570 raft_consensus.cc:348] T 00000000000000000000000000000000 P aecf840a944b4b159935a25676d1c54b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecf840a944b4b159935a25676d1c54b" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36757 } }
I20260924 03:27:16.428629 32570 raft_consensus.cc:374] T 00000000000000000000000000000000 P aecf840a944b4b159935a25676d1c54b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:27:16.428737 32570 raft_consensus.cc:729] T 00000000000000000000000000000000 P aecf840a944b4b159935a25676d1c54b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aecf840a944b4b159935a25676d1c54b, State: Initialized, Role: FOLLOWER
I20260924 03:27:16.429440 32570 consensus_queue.cc:260] T 00000000000000000000000000000000 P aecf840a944b4b159935a25676d1c54b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecf840a944b4b159935a25676d1c54b" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36757 } }
I20260924 03:27:16.429725 32570 raft_consensus.cc:388] T 00000000000000000000000000000000 P aecf840a944b4b159935a25676d1c54b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:27:16.429859 32570 raft_consensus.cc:482] T 00000000000000000000000000000000 P aecf840a944b4b159935a25676d1c54b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:27:16.430051 32570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aecf840a944b4b159935a25676d1c54b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:27:16.432904 32573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:16.425910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60271 (local address 127.10.12.190:36757)
0924 03:27:16.426120 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:16.426126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:16.426481 (+ 355us) server_negotiation.cc:407] Connection header received
0924 03:27:16.426665 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:16.426669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:16.426736 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:16.426820 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:16.427447 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:16.428242 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:16.429981 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:16.430578 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:16.430691 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:16.432092 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:16.432097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:16.432101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:27:16.432121 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:16.432163 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:27:16.432179 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:16.432252 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:16.432481 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 03:27:16.432564 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 03:27:16.433207 32570 raft_consensus.cc:504] T 00000000000000000000000000000000 P aecf840a944b4b159935a25676d1c54b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecf840a944b4b159935a25676d1c54b" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36757 } }
I20260924 03:27:16.433777 32570 leader_election.cc:302] T 00000000000000000000000000000000 P aecf840a944b4b159935a25676d1c54b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aecf840a944b4b159935a25676d1c54b; no voters:
I20260924 03:27:16.434387 32570 leader_election.cc:288] T 00000000000000000000000000000000 P aecf840a944b4b159935a25676d1c54b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:27:16.434526 32580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aecf840a944b4b159935a25676d1c54b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:27:16.435118 32580 raft_consensus.cc:686] T 00000000000000000000000000000000 P aecf840a944b4b159935a25676d1c54b [term 1 LEADER]: Becoming Leader. State: Replica: aecf840a944b4b159935a25676d1c54b, State: Running, Role: LEADER
I20260924 03:27:16.435663 32580 consensus_queue.cc:237] T 00000000000000000000000000000000 P aecf840a944b4b159935a25676d1c54b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecf840a944b4b159935a25676d1c54b" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36757 } }
I20260924 03:27:16.436509 32570 sys_catalog.cc:564] T 00000000000000000000000000000000 P aecf840a944b4b159935a25676d1c54b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:27:16.438082 32581 sys_catalog.cc:455] T 00000000000000000000000000000000 P aecf840a944b4b159935a25676d1c54b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aecf840a944b4b159935a25676d1c54b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecf840a944b4b159935a25676d1c54b" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36757 } } }
I20260924 03:27:16.438469 32581 sys_catalog.cc:458] T 00000000000000000000000000000000 P aecf840a944b4b159935a25676d1c54b [sys.catalog]: This master's current role is: LEADER
I20260924 03:27:16.439180 32582 sys_catalog.cc:455] T 00000000000000000000000000000000 P aecf840a944b4b159935a25676d1c54b [sys.catalog]: SysCatalogTable state changed. Reason: New leader aecf840a944b4b159935a25676d1c54b. Latest consensus state: current_term: 1 leader_uuid: "aecf840a944b4b159935a25676d1c54b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecf840a944b4b159935a25676d1c54b" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36757 } } }
I20260924 03:27:16.439531 32582 sys_catalog.cc:458] T 00000000000000000000000000000000 P aecf840a944b4b159935a25676d1c54b [sys.catalog]: This master's current role is: LEADER
I20260924 03:27:16.444566 32585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:27:16.450045 32585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:27:16.460573 32585 catalog_manager.cc:1347] Generated new cluster ID: 2346d423e4484082b190ee91acccccbc
I20260924 03:27:16.460713 32585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:27:16.482005 32585 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:27:16.484066 32585 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:27:16.497242 32585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aecf840a944b4b159935a25676d1c54b: Generated new TSK 0
I20260924 03:27:16.498427 32585 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:27:16.515614 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 32503
W20260924 03:27:16.516783 32565 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45061
W20260924 03:27:16.517531 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 32503: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:27:16.517670 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 32503
2026-09-24T03:27:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1863 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:27:16 dist-test-slave-00qs krb5kdc[32601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:27:16 dist-test-slave-00qs krb5kdc[32601](info): set up 2 sockets
Sep 24 03:27:16 dist-test-slave-00qs krb5kdc[32601](info): commencing operation
krb5kdc: starting...
W20260924 03:27:17.556780 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:27:17 dist-test-slave-00qs krb5kdc[32601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:27:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:27:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:27:17.823117 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:38561
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:36691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:38561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:27:18.166940 32614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:27:18.167251 32614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:27:18.167348 32614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:27:18.174906 32614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:27:18.175040 32614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:27:18.175087 32614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:27:18.175143 32614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:27:18.175186 32614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:27:18.185860 32614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:36691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:38561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:38561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:27:18.187171 32614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:27:18.188493 32614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:27:18.200415 32619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:18.200525 32620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:18.201283 32622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:27:18.201838 32614 server_base.cc:1034] running on GCE node
I20260924 03:27:18.202732 32614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:27:18.204397 32614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:27:18.205651 32614 hybrid_clock.cc:648] HybridClock initialized: now 1790220438205604 us; error 37 us; skew 500 ppm
I20260924 03:27:18.206027 32614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:27:18 dist-test-slave-00qs krb5kdc[32601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220438, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:27:18.209564 32614 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:27:18.211431 32614 webserver.cc:460] Webserver started at http://127.10.12.190:43985/ using document root <none> and password file <none>
I20260924 03:27:18.212220 32614 fs_manager.cc:359] Metadata directory not provided
I20260924 03:27:18.212347 32614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:27:18.212641 32614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:27:18.215811 32614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "f7cb901a295049a99d9b1ecf38872461"
format_stamp: "Formatted at 2026-09-24 03:27:18 on dist-test-slave-00qs"
server_key: "33d70dfd48a40df647d6c476c9d05b12"
server_key_iv: "297a293e5ab12d437c8119fdfefeaa98"
server_key_version: "encryptionkey@0"
I20260924 03:27:18.217010 32614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "f7cb901a295049a99d9b1ecf38872461"
format_stamp: "Formatted at 2026-09-24 03:27:18 on dist-test-slave-00qs"
server_key: "33d70dfd48a40df647d6c476c9d05b12"
server_key_iv: "297a293e5ab12d437c8119fdfefeaa98"
server_key_version: "encryptionkey@0"
I20260924 03:27:18.222838 32614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:27:18.226818 32629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:27:18.228564 32614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:27:18.228730 32614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "f7cb901a295049a99d9b1ecf38872461"
format_stamp: "Formatted at 2026-09-24 03:27:18 on dist-test-slave-00qs"
server_key: "33d70dfd48a40df647d6c476c9d05b12"
server_key_iv: "297a293e5ab12d437c8119fdfefeaa98"
server_key_version: "encryptionkey@0"
I20260924 03:27:18.228940 32614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:27:18.235440 32614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:27:18.239931 32614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:27:18.240232 32614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:27:18.259213 32614 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:38561
I20260924 03:27:18.259243 32680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:38561 every 8 connection(s)
I20260924 03:27:18.260746 32614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:27:18.264654 32681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:27:18.268702 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 32614
I20260924 03:27:18.268855 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:27:18.269291 10290 external_mini_cluster.cc:1428] Setting key 19fd27d7628e27dc6dfcee5ce3fa7138
I20260924 03:27:18.276527 32681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7cb901a295049a99d9b1ecf38872461: Bootstrap starting.
I20260924 03:27:18.278405 32684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:18.271331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43109 (local address 127.10.12.190:38561)
0924 03:27:18.272180 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:18.272192 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:18.272241 (+ 49us) server_negotiation.cc:407] Connection header received
0924 03:27:18.274814 (+ 2573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:18.274846 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:18.275246 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:18.275616 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:18.276787 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:18.276794 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:18.276808 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:27:18.276824 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:18.276894 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:27:18.276918 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:18.277096 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:18.277194 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 03:27:18.277457 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":344,"threads_started":1}
I20260924 03:27:18.279862 32681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7cb901a295049a99d9b1ecf38872461: Neither blocks nor log segments found. Creating new log.
I20260924 03:27:18.281034 32681 log.cc:824] T 00000000000000000000000000000000 P f7cb901a295049a99d9b1ecf38872461: Log is configured to *not* fsync() on all Append() calls
I20260924 03:27:18.282168 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:27:18.284132 32681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7cb901a295049a99d9b1ecf38872461: No bootstrap required, opened a new log
I20260924 03:27:18.289031 32684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:18.286390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50335 (local address 127.10.12.190:38561)
0924 03:27:18.286588 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:18.286594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:18.286988 (+ 394us) server_negotiation.cc:407] Connection header received
0924 03:27:18.287132 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:18.287138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:18.287205 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:18.287315 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:18.288257 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:18.288263 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:18.288268 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:27:18.288286 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:18.288335 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:27:18.288360 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:18.288486 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:18.288636 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 03:27:18.288705 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 03:27:18.292887 32681 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7cb901a295049a99d9b1ecf38872461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cb901a295049a99d9b1ecf38872461" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 38561 } }
I20260924 03:27:18.293283 32681 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7cb901a295049a99d9b1ecf38872461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:27:18.293380 32681 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7cb901a295049a99d9b1ecf38872461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7cb901a295049a99d9b1ecf38872461, State: Initialized, Role: FOLLOWER
I20260924 03:27:18.293964 32681 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7cb901a295049a99d9b1ecf38872461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cb901a295049a99d9b1ecf38872461" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 38561 } }
I20260924 03:27:18.294186 32681 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7cb901a295049a99d9b1ecf38872461 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:27:18.294315 32681 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7cb901a295049a99d9b1ecf38872461 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:27:18.294471 32681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7cb901a295049a99d9b1ecf38872461 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:27:18.296767 32681 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7cb901a295049a99d9b1ecf38872461 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cb901a295049a99d9b1ecf38872461" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 38561 } }
I20260924 03:27:18.297267 32681 leader_election.cc:302] T 00000000000000000000000000000000 P f7cb901a295049a99d9b1ecf38872461 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7cb901a295049a99d9b1ecf38872461; no voters:
I20260924 03:27:18.297875 32681 leader_election.cc:288] T 00000000000000000000000000000000 P f7cb901a295049a99d9b1ecf38872461 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:27:18.298062 32691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7cb901a295049a99d9b1ecf38872461 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:27:18.298699 32691 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7cb901a295049a99d9b1ecf38872461 [term 1 LEADER]: Becoming Leader. State: Replica: f7cb901a295049a99d9b1ecf38872461, State: Running, Role: LEADER
I20260924 03:27:18.299314 32691 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7cb901a295049a99d9b1ecf38872461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cb901a295049a99d9b1ecf38872461" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 38561 } }
I20260924 03:27:18.300019 32681 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7cb901a295049a99d9b1ecf38872461 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:27:18.302086 32693 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7cb901a295049a99d9b1ecf38872461 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7cb901a295049a99d9b1ecf38872461. Latest consensus state: current_term: 1 leader_uuid: "f7cb901a295049a99d9b1ecf38872461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cb901a295049a99d9b1ecf38872461" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 38561 } } }
I20260924 03:27:18.302425 32693 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7cb901a295049a99d9b1ecf38872461 [sys.catalog]: This master's current role is: LEADER
I20260924 03:27:18.302175 32692 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7cb901a295049a99d9b1ecf38872461 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7cb901a295049a99d9b1ecf38872461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cb901a295049a99d9b1ecf38872461" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 38561 } } }
I20260924 03:27:18.302533 32692 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7cb901a295049a99d9b1ecf38872461 [sys.catalog]: This master's current role is: LEADER
I20260924 03:27:18.303403 32698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:27:18.311378 32698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:27:18.326879 32698 catalog_manager.cc:1347] Generated new cluster ID: d37ab23794404958b74f75a548ad30ab
I20260924 03:27:18.327010 32698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:27:18.340224 32698 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:27:18.340719 32698 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:27:18.348542 32698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7cb901a295049a99d9b1ecf38872461: Generated new TSK 0
I20260924 03:27:18.349488 32698 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:27:18.359974 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 32614
W20260924 03:27:18.361159 32676 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43109
W20260924 03:27:18.361908 10290 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 32614: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:27:18.362080 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 32614
2026-09-24T03:27:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1843 ms)
[----------] 18 tests from AuthTokenIssuingTest (34324 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:27:18 dist-test-slave-00qs krb5kdc[32712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:27:18 dist-test-slave-00qs krb5kdc[32712](info): set up 2 sockets
Sep 24 03:27:18 dist-test-slave-00qs krb5kdc[32712](info): commencing operation
krb5kdc: starting...
W20260924 03:27:19.400671 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:27:19 dist-test-slave-00qs krb5kdc[32712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:27:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:27:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:27:19.668291 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:36195
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:34933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:36195,127.10.12.189:32993,127.10.12.188:41875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:27:20.000062 32725 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:27:20.000522 32725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:27:20.000638 32725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:27:20.000759 32725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:27:20.008550 32725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:27:20.008661 32725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:27:20.008735 32725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:27:20.008806 32725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:27:20.008886 32725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:27:20.019618 32725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:34933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:36195,127.10.12.189:32993,127.10.12.188:41875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:36195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:27:20.021018 32725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:27:20.022323 32725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:27:20.034281 32730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:20.034339 32733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:20.034348 32731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:27:20.034801 32725 server_base.cc:1034] running on GCE node
I20260924 03:27:20.035732 32725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:27:20.037256 32725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:27:20.038492 32725 hybrid_clock.cc:648] HybridClock initialized: now 1790220440038409 us; error 48 us; skew 500 ppm
I20260924 03:27:20.038830 32725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:27:20 dist-test-slave-00qs krb5kdc[32712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220440, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:27:20.042004 32725 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:27:20.043503 32725 webserver.cc:460] Webserver started at http://127.10.12.190:42777/ using document root <none> and password file <none>
I20260924 03:27:20.044260 32725 fs_manager.cc:359] Metadata directory not provided
I20260924 03:27:20.044353 32725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:27:20.044646 32725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:27:20.047742 32725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "3aceae61662c43309f33b2eb9b2c4698"
format_stamp: "Formatted at 2026-09-24 03:27:20 on dist-test-slave-00qs"
server_key: "a5513a4b73dec95979221a951620ad25"
server_key_iv: "3514455bc157437e055d094f4e39e07f"
server_key_version: "encryptionkey@0"
I20260924 03:27:20.048650 32725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "3aceae61662c43309f33b2eb9b2c4698"
format_stamp: "Formatted at 2026-09-24 03:27:20 on dist-test-slave-00qs"
server_key: "a5513a4b73dec95979221a951620ad25"
server_key_iv: "3514455bc157437e055d094f4e39e07f"
server_key_version: "encryptionkey@0"
I20260924 03:27:20.054945 32725 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 03:27:20.059687 32740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:27:20.061417 32725 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:27:20.061607 32725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "3aceae61662c43309f33b2eb9b2c4698"
format_stamp: "Formatted at 2026-09-24 03:27:20 on dist-test-slave-00qs"
server_key: "a5513a4b73dec95979221a951620ad25"
server_key_iv: "3514455bc157437e055d094f4e39e07f"
server_key_version: "encryptionkey@0"
I20260924 03:27:20.061818 32725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:27:20.084389 32725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:27:20.088973 32725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:27:20.089217 32725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:27:20.107295 32725 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:36195
I20260924 03:27:20.107302 323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:36195 every 8 connection(s)
I20260924 03:27:20.108676 32725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:27:20.112555 324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:27:20.114293 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 32725
I20260924 03:27:20.114522 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:27:20.115342 10290 external_mini_cluster.cc:1428] Setting key 8f7b106159f4e373530830bf3c0a870f
I20260924 03:27:20.121580 324 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36195 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:27:20 dist-test-slave-00qs krb5kdc[32712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220440, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.12.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.189@KRBTEST.COM" created.
I20260924 03:27:20.148612 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.125670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36195 (local address 127.0.0.1:42824)
0924 03:27:20.127767 (+ 2097us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.127829 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 03:27:20.130273 (+ 2444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:27:20.131360 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:27:20.131373 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:27:20.132180 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:27:20.132643 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:20.132668 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:20.134352 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:20.134357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:20.135977 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:20.135999 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:20.136989 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:20.136996 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:20.137166 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:20.137844 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:27:20.137877 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:27:20.140852 (+ 2975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:27:20.144297 (+ 3445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:20.144309 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:20.144322 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:20.144808 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:20.145294 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:20.145298 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:20.145300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:20.145480 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:20.146512 (+ 1032us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:27:20.146521 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:27:20.146754 (+ 233us) client_negotiation.cc:769] Sending connection context
0924 03:27:20.146985 (+ 231us) client_negotiation.cc:240] Negotiation successful
0924 03:27:20.147455 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1806,"mutex_wait_us":1,"thread_start_us":1558,"threads_started":1}
I20260924 03:27:20.148660 327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.125671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42824 (local address 127.10.12.190:36195)
0924 03:27:20.128370 (+ 2699us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:20.128376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:20.128418 (+ 42us) server_negotiation.cc:407] Connection header received
0924 03:27:20.130829 (+ 2411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:20.130836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:20.131108 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:20.131291 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:20.132898 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:20.134182 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:20.136132 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:20.136736 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:20.136880 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:20.141169 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:20.141178 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:20.141192 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:20.141230 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:20.144119 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:20.144990 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:20.144997 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:20.144999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:20.145161 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:20.145677 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:20.145681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:20.145683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:20.146316 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:20.146565 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:20.147175 (+ 610us) server_negotiation.cc:299] Negotiation successful
0924 03:27:20.147617 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":163,"server-negotiator.queue_time_us":2509,"spinlock_wait_cycles":16768,"thread_start_us":2294,"threads_started":1}
I20260924 03:27:20.151623 324 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:27:20.152877 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.152458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:32993 (local address 127.0.0.1:48208)
0924 03:27:20.152645 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.152759 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:32993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 03:27:20.153194 32742 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.12.189:32993: connect: Connection refused (error 111)
I20260924 03:27:20.155416 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.154938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:32993 (local address 127.0.0.1:48214)
0924 03:27:20.155135 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.155233 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:32993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 03:27:20.156005 324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.189:32993: Network error: Client connection negotiation failed: client connection to 127.10.12.189:32993: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.189.keytab.
Entry for principal kudu/127.10.12.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.189.keytab.
I20260924 03:27:20.180276 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.189:32993
--webserver_interface=127.10.12.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.189.keytab
--principal=kudu/127.10.12.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:34933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:36195,127.10.12.189:32993,127.10.12.188:41875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:27:20.205497 324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } attempt: 1
I20260924 03:27:20.206775 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.206176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:32993 (local address 127.0.0.1:48218)
0924 03:27:20.206415 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.206597 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:32993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 03:27:20.208062 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.207543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:32993 (local address 127.0.0.1:48220)
0924 03:27:20.207749 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.207849 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:32993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 03:27:20.208319 324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.189:32993: Network error: Client connection negotiation failed: client connection to 127.10.12.189:32993: connect: Connection refused (error 111)
I20260924 03:27:20.276630 324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } attempt: 2
I20260924 03:27:20.278048 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.277421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:32993 (local address 127.0.0.1:48222)
0924 03:27:20.277735 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.277859 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:32993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260924 03:27:20.279191 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.278755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:32993 (local address 127.0.0.1:48226)
0924 03:27:20.278922 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.279026 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:32993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 03:27:20.279443 324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.189:32993: Network error: Client connection negotiation failed: client connection to 127.10.12.189:32993: connect: Connection refused (error 111)
I20260924 03:27:20.370726 324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } attempt: 3
I20260924 03:27:20.371903 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.371480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:32993 (local address 127.0.0.1:48236)
0924 03:27:20.371674 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.371765 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:32993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 03:27:20.373000 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.372633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:32993 (local address 127.0.0.1:48250)
0924 03:27:20.372771 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.372843 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:32993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 03:27:20.373209 324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.189:32993: Network error: Client connection negotiation failed: client connection to 127.10.12.189:32993: connect: Connection refused (error 111)
I20260924 03:27:20.516443 324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } attempt: 4
W20260924 03:27:20.517002 331 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:27:20.517472 331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:27:20.517570 331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:27:20.517692 331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 03:27:20.517848 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.517213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:32993 (local address 127.0.0.1:48260)
0924 03:27:20.517593 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.517694 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:32993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260924 03:27:20.518849 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.518495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:32993 (local address 127.0.0.1:48264)
0924 03:27:20.518646 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.518728 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:32993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 03:27:20.519064 324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.189:32993: Network error: Client connection negotiation failed: client connection to 127.10.12.189:32993: connect: Connection refused (error 111)
W20260924 03:27:20.525604 331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:27:20.525705 331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:27:20.525776 331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:27:20.525838 331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:27:20.525902 331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:27:20.536406 331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:34933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:36195,127.10.12.189:32993,127.10.12.188:41875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.189.keytab
--principal=kudu/127.10.12.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.189:32993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.12.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:27:20.537771 331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:27:20.539134 331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:27:20.550781 336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:20.550802 337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:20.550731 339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:27:20.551649 331 server_base.cc:1034] running on GCE node
I20260924 03:27:20.552469 331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:27:20.553934 331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:27:20.555154 331 hybrid_clock.cc:648] HybridClock initialized: now 1790220440555088 us; error 33 us; skew 500 ppm
I20260924 03:27:20.555500 331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:27:20 dist-test-slave-00qs krb5kdc[32712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220440, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:27:20.558768 331 init.cc:377] Logged in from keytab as kudu/127.10.12.189@KRBTEST.COM (short username kudu)
I20260924 03:27:20.560426 331 webserver.cc:460] Webserver started at http://127.10.12.189:36755/ using document root <none> and password file <none>
I20260924 03:27:20.561101 331 fs_manager.cc:359] Metadata directory not provided
I20260924 03:27:20.561199 331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:27:20.561497 331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:27:20.564517 331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-1/data/instance:
uuid: "4ffbe4d187c24e89a46645bca6f69b0e"
format_stamp: "Formatted at 2026-09-24 03:27:20 on dist-test-slave-00qs"
server_key: "d8f409acd793a166a32d8adaed106574"
server_key_iv: "1db2e6df59a815c21bd00c77c49d512d"
server_key_version: "encryptionkey@0"
I20260924 03:27:20.565346 331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-1/wal/instance:
uuid: "4ffbe4d187c24e89a46645bca6f69b0e"
format_stamp: "Formatted at 2026-09-24 03:27:20 on dist-test-slave-00qs"
server_key: "d8f409acd793a166a32d8adaed106574"
server_key_iv: "1db2e6df59a815c21bd00c77c49d512d"
server_key_version: "encryptionkey@0"
I20260924 03:27:20.570739 331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:27:20.574631 346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:27:20.576164 331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 03:27:20.576329 331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-1/wal
uuid: "4ffbe4d187c24e89a46645bca6f69b0e"
format_stamp: "Formatted at 2026-09-24 03:27:20 on dist-test-slave-00qs"
server_key: "d8f409acd793a166a32d8adaed106574"
server_key_iv: "1db2e6df59a815c21bd00c77c49d512d"
server_key_version: "encryptionkey@0"
I20260924 03:27:20.576540 331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:27:20.591343 331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:27:20.595698 331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:27:20.596029 331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:27:20.614502 331 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.189:32993
I20260924 03:27:20.614506 397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.189:32993 every 8 connection(s)
I20260924 03:27:20.615932 331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-1/data/info.pb
I20260924 03:27:20.616497 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 331
I20260924 03:27:20.616706 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-1/wal/instance
I20260924 03:27:20.617262 10290 external_mini_cluster.cc:1428] Setting key f2de2386fdb98b4c8907a0f0c73a4f5e
I20260924 03:27:20.620167 398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:27:20.626881 398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36195 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.12.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.188@KRBTEST.COM" created.
Sep 24 03:27:20 dist-test-slave-00qs krb5kdc[32712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220440, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.189@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:27:20.648218 327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.629699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42832 (local address 127.10.12.190:36195)
0924 03:27:20.629905 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:20.629910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:20.631312 (+ 1402us) server_negotiation.cc:407] Connection header received
0924 03:27:20.633351 (+ 2039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:20.633367 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:20.633450 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:20.633536 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:20.635196 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:20.635898 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:20.637505 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:20.637933 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:20.638045 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:20.642341 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:20.642347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:20.642350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:20.642388 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:20.645084 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:20.645869 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:20.645873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:20.645876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:20.646007 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:20.646430 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:20.646434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:20.646436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:20.646730 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:20.646884 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:20.647630 (+ 746us) server_negotiation.cc:299] Negotiation successful
0924 03:27:20.647871 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 03:27:20.649230 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.630276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36195 (local address 127.0.0.1:42832)
0924 03:27:20.631128 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.631173 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:27:20.633047 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:27:20.633802 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:27:20.633811 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:27:20.634375 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:27:20.634996 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:20.635015 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:20.636084 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:20.636089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:20.637335 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:20.637349 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:20.638167 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:20.638175 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:20.638416 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:20.639053 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:27:20.639076 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:27:20.642078 (+ 3002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:27:20.645250 (+ 3172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:20.645262 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:20.645278 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:20.645737 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:20.646117 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:20.646122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:20.646126 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:20.646315 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:20.646877 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:27:20.646885 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:27:20.647259 (+ 374us) client_negotiation.cc:769] Sending connection context
0924 03:27:20.647514 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 03:27:20.648093 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":340,"threads_started":1}
I20260924 03:27:20.651064 398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:27:20 dist-test-slave-00qs krb5kdc[32712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220440, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.189@KRBTEST.COM for kudu/127.10.12.189@KRBTEST.COM
I20260924 03:27:20.667793 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.651932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:32993 (local address 127.0.0.1:48268)
0924 03:27:20.652132 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.652153 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:27:20.652263 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:27:20.653549 (+ 1286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:27:20.653554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:27:20.653986 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:27:20.654145 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:20.654154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:20.655997 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:20.656004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:20.656860 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:20.656872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:20.658034 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:20.658039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:20.658176 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:20.658779 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:27:20.658796 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:27:20.661027 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:27:20.665128 (+ 4101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:20.665133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:20.665136 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:20.665536 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:20.665924 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:20.665928 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:20.665932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:20.666111 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:20.666821 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:27:20.666825 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:27:20.666969 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 03:27:20.667080 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 03:27:20.667369 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 03:27:20.667835 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.652229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48268 (local address 127.10.12.189:32993)
0924 03:27:20.652960 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:20.652968 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:20.652997 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:27:20.653049 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:20.653057 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:20.653289 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:20.653590 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:20.654318 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:20.655810 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:20.656995 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:20.657797 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:20.657924 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:20.661283 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:20.661302 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:20.661321 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:20.661362 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:20.664985 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:20.665669 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:20.665675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:20.665678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:20.665798 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:20.666244 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:20.666247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:20.666249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:20.666683 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:20.666880 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:20.667249 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 03:27:20.667469 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":409,"threads_started":1}
I20260924 03:27:20.670472 398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:27:20.671666 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.671118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:41875 (local address 127.0.0.1:41322)
0924 03:27:20.671336 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.671499 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260924 03:27:20.671926 349 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
I20260924 03:27:20.673583 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.673211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:41875 (local address 127.0.0.1:41328)
0924 03:27:20.673351 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.673433 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 03:27:20.673820 398 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:41875: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.188.keytab.
Entry for principal kudu/127.10.12.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.188.keytab.
I20260924 03:27:20.675521 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.188:41875
--webserver_interface=127.10.12.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.188.keytab
--principal=kudu/127.10.12.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:34933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:36195,127.10.12.189:32993,127.10.12.188:41875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:27:20.723187 398 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } attempt: 1
I20260924 03:27:20.724748 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.724128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:41875 (local address 127.0.0.1:41332)
0924 03:27:20.724372 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.724532 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 03:27:20.726104 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.725630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:41875 (local address 127.0.0.1:41340)
0924 03:27:20.725808 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.725935 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 03:27:20.726394 398 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:41875: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
I20260924 03:27:20.794633 398 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } attempt: 2
I20260924 03:27:20.795907 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.795373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:41875 (local address 127.0.0.1:41344)
0924 03:27:20.795648 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.795738 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260924 03:27:20.796995 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.796668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:41875 (local address 127.0.0.1:41360)
0924 03:27:20.796802 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.796879 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 03:27:20.797199 398 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:41875: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
I20260924 03:27:20.818316 324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } attempt: 5
Sep 24 03:27:20 dist-test-slave-00qs krb5kdc[32712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220440, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for kudu/127.10.12.189@KRBTEST.COM
I20260924 03:27:20.832845 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.819059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:32993 (local address 127.0.0.1:48270)
0924 03:27:20.819246 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.819267 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:27:20.819381 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:27:20.819770 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:27:20.819775 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:27:20.820223 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:27:20.820428 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:20.820438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:20.821438 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:20.821443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:20.822286 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:20.822299 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:20.823045 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:20.823051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:20.823273 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:20.823874 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:27:20.823893 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:27:20.826252 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:27:20.830131 (+ 3879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:20.830141 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:20.830145 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:20.830747 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:20.831221 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:20.831225 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:20.831227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:20.831390 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:20.832037 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:27:20.832041 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:27:20.832187 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 03:27:20.832317 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 03:27:20.832567 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 03:27:20.833051 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.819263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48270 (local address 127.10.12.189:32993)
0924 03:27:20.819476 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:20.819482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:20.819499 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:27:20.819555 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:20.819559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:20.819635 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:20.819719 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:20.820586 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:20.821293 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:20.822444 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:20.822894 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:20.823012 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:20.826542 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:20.826550 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:20.826555 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:20.826602 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:20.829919 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:20.830896 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:20.830902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:20.830905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:20.831077 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:20.831502 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:20.831507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:20.831511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:20.831877 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:20.832069 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:20.832437 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 03:27:20.832693 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 03:27:20.834693 324 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:27:20.835785 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.835388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:41875 (local address 127.0.0.1:41372)
0924 03:27:20.835560 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.835640 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 03:27:20.836769 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.836438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:41875 (local address 127.0.0.1:41386)
0924 03:27:20.836581 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.836669 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 03:27:20.836966 324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:41875: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
I20260924 03:27:20.888216 324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } attempt: 1
I20260924 03:27:20.888427 398 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } attempt: 3
I20260924 03:27:20.889550 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.888981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:41875 (local address 127.0.0.1:41396)
0924 03:27:20.889228 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.889346 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 03:27:20.889562 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.889022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:41875 (local address 127.0.0.1:41394)
0924 03:27:20.889298 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.889383 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260924 03:27:20.890708 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.890285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:41875 (local address 127.0.0.1:41406)
0924 03:27:20.890476 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.890568 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 03:27:20.890841 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.890271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:41875 (local address 127.0.0.1:41402)
0924 03:27:20.890615 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.890704 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208}
W20260924 03:27:20.891054 324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:41875: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
W20260924 03:27:20.891093 398 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:41875: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
I20260924 03:27:20.959311 324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } attempt: 2
I20260924 03:27:20.960556 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.960077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:41875 (local address 127.0.0.1:41410)
0924 03:27:20.960286 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.960390 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 03:27:20.961663 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:20.961292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:41875 (local address 127.0.0.1:41418)
0924 03:27:20.961463 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:20.961546 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 03:27:20.961881 324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:41875: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
W20260924 03:27:21.011706 405 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:27:21.012117 405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:27:21.012207 405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:27:21.012291 405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:27:21.019922 405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:27:21.020076 405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:27:21.020139 405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:27:21.020188 405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:27:21.020249 405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:27:21.030753 405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:34933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:36195,127.10.12.189:32993,127.10.12.188:41875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.188.keytab
--principal=kudu/127.10.12.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.188:41875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.12.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:27:21.032158 405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:27:21.033468 405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:27:21.034322 398 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } attempt: 4
I20260924 03:27:21.035410 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.034944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:41875 (local address 127.0.0.1:41428)
0924 03:27:21.035174 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:21.035258 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 03:27:21.036528 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.036186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:41875 (local address 127.0.0.1:41436)
0924 03:27:21.036327 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:21.036401 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 03:27:21.036751 398 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:41875: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
W20260924 03:27:21.045814 413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:21.045892 410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:21.045847 411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:27:21.047578 405 server_base.cc:1034] running on GCE node
I20260924 03:27:21.048530 405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:27:21.050112 405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:27:21.051357 405 hybrid_clock.cc:648] HybridClock initialized: now 1790220441051262 us; error 79 us; skew 500 ppm
I20260924 03:27:21.051687 405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:27:21 dist-test-slave-00qs krb5kdc[32712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220441, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:27:21.055006 405 init.cc:377] Logged in from keytab as kudu/127.10.12.188@KRBTEST.COM (short username kudu)
I20260924 03:27:21.056636 405 webserver.cc:460] Webserver started at http://127.10.12.188:36815/ using document root <none> and password file <none>
I20260924 03:27:21.057348 405 fs_manager.cc:359] Metadata directory not provided
I20260924 03:27:21.057446 405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:27:21.057704 405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:27:21.060807 405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-2/data/instance:
uuid: "e65f92a1043d46268f529187fa528e15"
format_stamp: "Formatted at 2026-09-24 03:27:21 on dist-test-slave-00qs"
server_key: "495abdd20efceeda64bcbee9d56c302b"
server_key_iv: "3ea9d0a32b73956950e0c918c3278c3b"
server_key_version: "encryptionkey@0"
I20260924 03:27:21.061623 405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-2/wal/instance:
uuid: "e65f92a1043d46268f529187fa528e15"
format_stamp: "Formatted at 2026-09-24 03:27:21 on dist-test-slave-00qs"
server_key: "495abdd20efceeda64bcbee9d56c302b"
server_key_iv: "3ea9d0a32b73956950e0c918c3278c3b"
server_key_version: "encryptionkey@0"
I20260924 03:27:21.067233 405 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:27:21.068120 324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } attempt: 3
I20260924 03:27:21.069341 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.068800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:41875 (local address 127.0.0.1:41446)
0924 03:27:21.069051 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:21.069147 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 03:27:21.070559 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.070067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:41875 (local address 127.0.0.1:41456)
0924 03:27:21.070276 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:21.070375 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260924 03:27:21.070793 324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:41875: Network error: Client connection negotiation failed: client connection to 127.10.12.188:41875: connect: Connection refused (error 111)
I20260924 03:27:21.071399 420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:27:21.072868 405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:27:21.073061 405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-2/wal
uuid: "e65f92a1043d46268f529187fa528e15"
format_stamp: "Formatted at 2026-09-24 03:27:21 on dist-test-slave-00qs"
server_key: "495abdd20efceeda64bcbee9d56c302b"
server_key_iv: "3ea9d0a32b73956950e0c918c3278c3b"
server_key_version: "encryptionkey@0"
I20260924 03:27:21.073284 405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:27:21.088348 405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:27:21.093477 405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:27:21.093788 405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:27:21.111981 405 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.188:41875
I20260924 03:27:21.111940 471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.188:41875 every 8 connection(s)
I20260924 03:27:21.113839 405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-2/data/info.pb
I20260924 03:27:21.117880 472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:27:21.120574 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 405
I20260924 03:27:21.120728 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790220067268388-10290-0/minicluster-data/master-2/wal/instance
I20260924 03:27:21.121158 10290 external_mini_cluster.cc:1428] Setting key 637097f824d6c4f04e9694c3ff461a01
I20260924 03:27:21.126039 472 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36195 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:27:21 dist-test-slave-00qs krb5kdc[32712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.188@KRBTEST.COM
Sep 24 03:27:21 dist-test-slave-00qs krb5kdc[32712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220441, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.188@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:27:21.145514 327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.129355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42836 (local address 127.10.12.190:36195)
0924 03:27:21.129553 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:21.129558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:21.130157 (+ 599us) server_negotiation.cc:407] Connection header received
0924 03:27:21.130360 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:21.130364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:21.130425 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:21.130509 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:21.131895 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.132782 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.134617 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.135100 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.135206 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:21.138850 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:21.138857 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:21.138862 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:21.138903 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:21.142105 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:21.142927 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:21.142933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:21.142936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:21.143100 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:21.143523 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:21.143528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:21.143531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:21.143901 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:21.144098 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:21.144870 (+ 772us) server_negotiation.cc:299] Negotiation successful
0924 03:27:21.145159 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 03:27:21.146538 475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.123228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41470 (local address 127.10.12.188:41875)
0924 03:27:21.124108 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:21.124124 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:21.124177 (+ 53us) server_negotiation.cc:407] Connection header received
0924 03:27:21.126387 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:21.126417 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:21.127009 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:21.127471 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:21.129039 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.130878 (+ 1839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.132484 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.133394 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.133533 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:21.138216 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:21.138225 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:21.138242 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:21.138277 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:21.141871 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:21.142806 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:21.142813 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:21.142816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:21.143039 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:21.143478 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:21.143483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:21.143486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:21.144213 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:21.144513 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:21.144886 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 03:27:21.145230 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":606,"thread_start_us":391,"threads_started":1}
I20260924 03:27:21.146576 476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.129284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:36195 (local address 127.0.0.1:42836)
0924 03:27:21.130004 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:21.130053 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 03:27:21.130220 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:27:21.130587 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:27:21.130594 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:27:21.131407 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:27:21.131724 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:21.131737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.132975 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.132979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:21.134399 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:21.134416 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.135359 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.135368 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:21.135560 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:21.136244 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:27:21.136276 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:27:21.138595 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:27:21.142234 (+ 3639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:21.142249 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:21.142262 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:21.142789 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:21.143214 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:21.143218 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:21.143222 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:21.143409 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:21.144108 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:27:21.144116 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:27:21.144446 (+ 330us) client_negotiation.cc:769] Sending connection context
0924 03:27:21.144737 (+ 291us) client_negotiation.cc:240] Negotiation successful
0924 03:27:21.145513 (+ 776us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":306,"thread_start_us":343,"threads_started":1}
I20260924 03:27:21.148224 472 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:27:21 dist-test-slave-00qs krb5kdc[32712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220441, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.188@KRBTEST.COM for kudu/127.10.12.189@KRBTEST.COM
Sep 24 03:27:21 dist-test-slave-00qs krb5kdc[32712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:27:21.162480 476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.148909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:32993 (local address 127.0.0.1:48274)
0924 03:27:21.149073 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:21.149093 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:27:21.149201 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:27:21.149544 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:27:21.149548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:27:21.149987 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:27:21.150189 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:21.150202 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.151169 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.151175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:21.152132 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:21.152146 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.152845 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.152850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:21.153062 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:21.153697 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:27:21.153722 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:27:21.156050 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:27:21.159746 (+ 3696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:21.159753 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:21.159758 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:21.160375 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:21.160834 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:21.160839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:21.160843 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:21.161047 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:21.161741 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:27:21.161745 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:27:21.161898 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 03:27:21.162014 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 03:27:21.162238 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 03:27:21.162770 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.149036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48274 (local address 127.10.12.189:32993)
0924 03:27:21.149253 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:21.149259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:21.149278 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:27:21.149340 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:21.149344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:21.149415 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:21.149489 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:21.150340 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.151036 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.152271 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.152704 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.152817 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:21.156250 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:21.156255 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:21.156259 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:21.156294 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:21.159580 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:21.160523 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:21.160529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:21.160533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:21.160721 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:21.161190 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:21.161196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:21.161200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:21.161597 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:21.161775 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:21.162171 (+ 396us) server_negotiation.cc:299] Negotiation successful
0924 03:27:21.162396 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 03:27:21.163733 472 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:27:21.165997 327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.149898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42838 (local address 127.10.12.190:36195)
0924 03:27:21.150048 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:21.150051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:21.150066 (+ 15us) server_negotiation.cc:407] Connection header received
0924 03:27:21.150205 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:21.150208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:21.150260 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:21.150348 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:21.151526 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.152279 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.153625 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.154250 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.154381 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:21.159161 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:21.159168 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:21.159173 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:21.159218 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:21.162831 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:21.163706 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:21.163712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:21.163715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:21.163888 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:21.164411 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:21.164416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:21.164419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:21.164830 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:21.165101 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:21.165419 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 03:27:21.165649 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 03:27:21 dist-test-slave-00qs krb5kdc[32712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220441, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.188@KRBTEST.COM for kudu/127.10.12.188@KRBTEST.COM
Sep 24 03:27:21 dist-test-slave-00qs krb5kdc[32712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.189@KRBTEST.COM
I20260924 03:27:21.177349 475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.164680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41474 (local address 127.10.12.188:41875)
0924 03:27:21.164813 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:21.164816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:21.164833 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:27:21.164871 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:21.164874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:21.164942 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:21.165032 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:21.165903 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.166695 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.168495 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.168891 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.169009 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:21.172163 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:21.172170 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:21.172174 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:21.172203 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:21.174623 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:21.175446 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:21.175451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:21.175454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:21.175606 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:21.175986 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:21.175992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:21.175995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:21.176270 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:21.176398 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:21.176766 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 03:27:21.177009 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 03:27:21.177610 476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.164503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:41875 (local address 127.0.0.1:41474)
0924 03:27:21.164671 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:21.164690 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:27:21.164786 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:27:21.165090 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:27:21.165095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:27:21.165555 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:27:21.165722 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:21.165734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.166809 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.166815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:21.168373 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:21.168386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.169023 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.169029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:21.169225 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:21.169824 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:27:21.169846 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:27:21.171945 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:27:21.174748 (+ 2803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:21.174754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:21.174759 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:21.175271 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:21.175715 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:21.175720 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:21.175724 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:21.175855 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:21.176379 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:27:21.176383 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:27:21.176535 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 03:27:21.176673 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 03:27:21.177269 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 03:27:21.182412 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.167708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48284 (local address 127.10.12.189:32993)
0924 03:27:21.167860 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:21.167864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:21.167878 (+ 14us) server_negotiation.cc:407] Connection header received
0924 03:27:21.167920 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:21.167923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:21.168032 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:21.168110 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:21.169297 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.170299 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.171526 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.172012 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.172132 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:21.176209 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:21.176214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:21.176218 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:21.176253 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:21.179107 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:21.180024 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:21.180030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:21.180033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:21.180200 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:21.180648 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:21.180653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:21.180657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:21.181040 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:21.181318 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:21.181676 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 03:27:21.181917 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 03:27:21.183521 472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e65f92a1043d46268f529187fa528e15: Bootstrap starting.
I20260924 03:27:21.187533 472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e65f92a1043d46268f529187fa528e15: Neither blocks nor log segments found. Creating new log.
I20260924 03:27:21.187917 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:27:21.188927 472 log.cc:824] T 00000000000000000000000000000000 P e65f92a1043d46268f529187fa528e15: Log is configured to *not* fsync() on all Append() calls
I20260924 03:27:21.192265 472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e65f92a1043d46268f529187fa528e15: No bootstrap required, opened a new log
I20260924 03:27:21.191303 32767 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3aceae61662c43309f33b2eb9b2c4698"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42838
I20260924 03:27:21.199739 472 raft_consensus.cc:348] T 00000000000000000000000000000000 P e65f92a1043d46268f529187fa528e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aceae61662c43309f33b2eb9b2c4698" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36195 } } peers { permanent_uuid: "4ffbe4d187c24e89a46645bca6f69b0e" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } } peers { permanent_uuid: "e65f92a1043d46268f529187fa528e15" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } }
I20260924 03:27:21.200166 472 raft_consensus.cc:729] T 00000000000000000000000000000000 P e65f92a1043d46268f529187fa528e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e65f92a1043d46268f529187fa528e15, State: Initialized, Role: FOLLOWER
I20260924 03:27:21.200644 472 consensus_queue.cc:260] T 00000000000000000000000000000000 P e65f92a1043d46268f529187fa528e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aceae61662c43309f33b2eb9b2c4698" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36195 } } peers { permanent_uuid: "4ffbe4d187c24e89a46645bca6f69b0e" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } } peers { permanent_uuid: "e65f92a1043d46268f529187fa528e15" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } }
I20260924 03:27:21.201716 478 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65f92a1043d46268f529187fa528e15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aceae61662c43309f33b2eb9b2c4698" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36195 } } peers { permanent_uuid: "4ffbe4d187c24e89a46645bca6f69b0e" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } } peers { permanent_uuid: "e65f92a1043d46268f529187fa528e15" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } } }
I20260924 03:27:21.202096 478 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65f92a1043d46268f529187fa528e15 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:27:21.202916 472 sys_catalog.cc:564] T 00000000000000000000000000000000 P e65f92a1043d46268f529187fa528e15 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 03:27:21.213131 489 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e65f92a1043d46268f529187fa528e15: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:27:21.213291 489 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:27:21.210536 405 master_runner.cc:186] Error getting master registration for 127.10.12.190:36195: OK, instance_id { permanent_uuid: "3aceae61662c43309f33b2eb9b2c4698" instance_seqno: 1790220440102119 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:27:21.222074 405 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:27:21.248037 324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } attempt: 4
Sep 24 03:27:21 dist-test-slave-00qs krb5kdc[32712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220440, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for kudu/127.10.12.188@KRBTEST.COM
I20260924 03:27:21.264467 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.248828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:41875 (local address 127.0.0.1:41486)
0924 03:27:21.249067 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:21.249089 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:27:21.249248 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:27:21.249601 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:27:21.249607 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:27:21.250157 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:27:21.250394 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:21.250408 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.251946 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.251979 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:21.253503 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:21.253521 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.254586 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.254592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:21.254810 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:21.255545 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:27:21.255571 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:27:21.258379 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:27:21.261630 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:21.261637 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:21.261642 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:21.262080 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:21.262526 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:21.262531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:21.262535 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:21.262685 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:21.263300 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:27:21.263305 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:27:21.263488 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 03:27:21.263615 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 03:27:21.264060 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 03:27:21.264475 475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.248907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41486 (local address 127.10.12.188:41875)
0924 03:27:21.249105 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:21.249114 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:21.249159 (+ 45us) server_negotiation.cc:407] Connection header received
0924 03:27:21.249377 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:21.249381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:21.249455 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:21.249549 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:21.250559 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.251767 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.253696 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.254418 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.254541 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:21.258605 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:21.258611 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:21.258614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:21.258651 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:21.261450 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:21.262250 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:21.262254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:21.262257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:21.262393 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:21.262804 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:21.262808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:21.262810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:21.263172 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:21.263348 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:21.263741 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 03:27:21.264105 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:27:21.270985 324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698: Bootstrap starting.
I20260924 03:27:21.275075 324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698: Neither blocks nor log segments found. Creating new log.
I20260924 03:27:21.276437 324 log.cc:824] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698: Log is configured to *not* fsync() on all Append() calls
I20260924 03:27:21.280539 324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698: No bootstrap required, opened a new log
I20260924 03:27:21.288394 324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aceae61662c43309f33b2eb9b2c4698" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36195 } } peers { permanent_uuid: "4ffbe4d187c24e89a46645bca6f69b0e" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } } peers { permanent_uuid: "e65f92a1043d46268f529187fa528e15" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } }
I20260924 03:27:21.288844 324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aceae61662c43309f33b2eb9b2c4698, State: Initialized, Role: FOLLOWER
I20260924 03:27:21.289368 324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aceae61662c43309f33b2eb9b2c4698" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36195 } } peers { permanent_uuid: "4ffbe4d187c24e89a46645bca6f69b0e" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } } peers { permanent_uuid: "e65f92a1043d46268f529187fa528e15" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } }
I20260924 03:27:21.290477 492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aceae61662c43309f33b2eb9b2c4698" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36195 } } peers { permanent_uuid: "4ffbe4d187c24e89a46645bca6f69b0e" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } } peers { permanent_uuid: "e65f92a1043d46268f529187fa528e15" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } } }
I20260924 03:27:21.290886 492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:27:21.291711 324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:27:21.291777 32767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 03:27:21.292099 32767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:27:21.296258 32767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aceae61662c43309f33b2eb9b2c4698" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36195 } } peers { permanent_uuid: "4ffbe4d187c24e89a46645bca6f69b0e" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } } peers { permanent_uuid: "e65f92a1043d46268f529187fa528e15" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } }
W20260924 03:27:21.302876 503 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:27:21.303033 503 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:27:21.303980 32767 leader_election.cc:288] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698 [CANDIDATE]: Term 1 election: Requested vote from peers 4ffbe4d187c24e89a46645bca6f69b0e (127.10.12.189:32993), e65f92a1043d46268f529187fa528e15 (127.10.12.188:41875)
I20260924 03:27:21.304455 447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3aceae61662c43309f33b2eb9b2c4698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e65f92a1043d46268f529187fa528e15"
I20260924 03:27:21.304976 447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e65f92a1043d46268f529187fa528e15 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:27:21.308801 447 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e65f92a1043d46268f529187fa528e15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aceae61662c43309f33b2eb9b2c4698 in term 1.
I20260924 03:27:21.309729 32743 leader_election.cc:302] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aceae61662c43309f33b2eb9b2c4698, e65f92a1043d46268f529187fa528e15; no voters:
I20260924 03:27:21.310303 492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:27:21.324249 492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698 [term 1 LEADER]: Becoming Leader. State: Replica: 3aceae61662c43309f33b2eb9b2c4698, State: Running, Role: LEADER
I20260924 03:27:21.325549 492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aceae61662c43309f33b2eb9b2c4698" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36195 } } peers { permanent_uuid: "4ffbe4d187c24e89a46645bca6f69b0e" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } } peers { permanent_uuid: "e65f92a1043d46268f529187fa528e15" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } }
I20260924 03:27:21.302008 32725 master_runner.cc:186] Error getting master registration for 127.10.12.189:32993: OK, instance_id { permanent_uuid: "4ffbe4d187c24e89a46645bca6f69b0e" instance_seqno: 1790220440609211 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:27:21.332167 32725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:27:21.335980 398 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } attempt: 5
W20260924 03:27:21.345346 32746 tablet.cc:2369] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:27:21.346592 426 tablet.cc:2369] T 00000000000000000000000000000000 P e65f92a1043d46268f529187fa528e15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:27:21.349812 505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3aceae61662c43309f33b2eb9b2c4698. Latest consensus state: current_term: 1 leader_uuid: "3aceae61662c43309f33b2eb9b2c4698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aceae61662c43309f33b2eb9b2c4698" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36195 } } peers { permanent_uuid: "4ffbe4d187c24e89a46645bca6f69b0e" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } } peers { permanent_uuid: "e65f92a1043d46268f529187fa528e15" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } } }
I20260924 03:27:21.350296 505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698 [sys.catalog]: This master's current role is: LEADER
Sep 24 03:27:21 dist-test-slave-00qs krb5kdc[32712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220440, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.189@KRBTEST.COM for kudu/127.10.12.188@KRBTEST.COM
I20260924 03:27:21.361264 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.336760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:41875 (local address 127.0.0.1:41492)
0924 03:27:21.337186 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:21.337208 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:27:21.337341 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:27:21.338468 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:27:21.338474 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:27:21.339033 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:27:21.339276 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:21.339290 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.340979 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.340986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:21.342512 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:21.342531 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.344091 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.344098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:21.344383 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:21.349969 (+ 5586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:27:21.349998 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:27:21.353283 (+ 3285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:27:21.358014 (+ 4731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:21.358021 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:21.358026 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:21.358652 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:21.359194 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:21.359200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:21.359205 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:21.359394 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:21.360159 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:27:21.360165 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:27:21.360341 (+ 176us) client_negotiation.cc:769] Sending connection context
0924 03:27:21.360508 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 03:27:21.360855 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285}
I20260924 03:27:21.362612 475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.337747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41492 (local address 127.10.12.188:41875)
0924 03:27:21.337952 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:21.337959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:21.337977 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:27:21.338046 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:21.338052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:21.338134 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:21.338240 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:21.339502 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.340790 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.342916 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.343700 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.343826 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:21.353687 (+ 9861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:21.353695 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:21.353700 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:21.353754 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:21.357760 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:21.358833 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:21.358839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:21.358842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:21.359041 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:21.359548 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:21.359553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:21.359556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:21.359921 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:21.360093 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:21.361566 (+ 1473us) server_negotiation.cc:299] Negotiation successful
0924 03:27:21.362132 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 03:27:21.370326 398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ffbe4d187c24e89a46645bca6f69b0e: Bootstrap starting.
I20260924 03:27:21.374281 513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:27:21.376833 398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ffbe4d187c24e89a46645bca6f69b0e: Neither blocks nor log segments found. Creating new log.
I20260924 03:27:21.379020 513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:27:21.382548 398 log.cc:824] T 00000000000000000000000000000000 P 4ffbe4d187c24e89a46645bca6f69b0e: Log is configured to *not* fsync() on all Append() calls
I20260924 03:27:21.389605 398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ffbe4d187c24e89a46645bca6f69b0e: No bootstrap required, opened a new log
I20260924 03:27:21.409452 447 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e65f92a1043d46268f529187fa528e15 [term 1 FOLLOWER]: Refusing update from remote peer 3aceae61662c43309f33b2eb9b2c4698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:27:21.410656 505 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e65f92a1043d46268f529187fa528e15" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:27:21.441339 398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ffbe4d187c24e89a46645bca6f69b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aceae61662c43309f33b2eb9b2c4698" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36195 } } peers { permanent_uuid: "4ffbe4d187c24e89a46645bca6f69b0e" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } } peers { permanent_uuid: "e65f92a1043d46268f529187fa528e15" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } }
I20260924 03:27:21.441874 398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ffbe4d187c24e89a46645bca6f69b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ffbe4d187c24e89a46645bca6f69b0e, State: Initialized, Role: FOLLOWER
I20260924 03:27:21.442476 398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ffbe4d187c24e89a46645bca6f69b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aceae61662c43309f33b2eb9b2c4698" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36195 } } peers { permanent_uuid: "4ffbe4d187c24e89a46645bca6f69b0e" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } } peers { permanent_uuid: "e65f92a1043d46268f529187fa528e15" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } }
I20260924 03:27:21.472338 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.350912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48292 (local address 127.10.12.189:32993)
0924 03:27:21.351111 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:21.351118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:21.411108 (+ 59990us) server_negotiation.cc:407] Connection header received
0924 03:27:21.412234 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:21.412240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:21.412330 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:21.412486 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:21.429575 (+ 17089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.430759 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.438453 (+ 7694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.439294 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.439448 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:21.443456 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:21.443465 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:21.443471 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:21.443526 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:21.449894 (+ 6368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:21.452158 (+ 2264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:21.452166 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:21.452170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:21.452805 (+ 635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:21.453764 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:21.453771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:21.453774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:21.454721 (+ 947us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:21.456821 (+ 2100us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:21.457859 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0924 03:27:21.469993 (+ 12134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 03:27:21.471864 478 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65f92a1043d46268f529187fa528e15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3aceae61662c43309f33b2eb9b2c4698. Latest consensus state: current_term: 1 leader_uuid: "3aceae61662c43309f33b2eb9b2c4698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aceae61662c43309f33b2eb9b2c4698" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36195 } } peers { permanent_uuid: "4ffbe4d187c24e89a46645bca6f69b0e" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } } peers { permanent_uuid: "e65f92a1043d46268f529187fa528e15" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } } }
I20260924 03:27:21.472719 478 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65f92a1043d46268f529187fa528e15 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:27:21.473687 475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.409862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41506 (local address 127.10.12.188:41875)
0924 03:27:21.410087 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:21.410094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:21.416862 (+ 6768us) server_negotiation.cc:407] Connection header received
0924 03:27:21.417345 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:21.417351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:21.417452 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:21.417527 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:21.421089 (+ 3562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.422521 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.458424 (+ 35903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.459246 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.459416 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:21.464228 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:21.464237 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:21.464243 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:21.464300 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:21.468732 (+ 4432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:21.469827 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:21.469834 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:21.469844 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:21.470043 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:21.470632 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:21.470669 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:21.470673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:21.471606 (+ 933us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:21.471903 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:21.472780 (+ 877us) server_negotiation.cc:299] Negotiation successful
0924 03:27:21.473088 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 03:27:21.475212 519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ffbe4d187c24e89a46645bca6f69b0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aceae61662c43309f33b2eb9b2c4698" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36195 } } peers { permanent_uuid: "4ffbe4d187c24e89a46645bca6f69b0e" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } } peers { permanent_uuid: "e65f92a1043d46268f529187fa528e15" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } } }
I20260924 03:27:21.475617 519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ffbe4d187c24e89a46645bca6f69b0e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:27:21.477814 515 mvcc.cc:205] Tried to move back new op lower bound from 7332742928001286144 to 7332742927681122304. Current Snapshot: MvccSnapshot[applied={T|T < 7332742928001286144}]
I20260924 03:27:21.478631 398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ffbe4d187c24e89a46645bca6f69b0e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:27:21.480254 372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ffbe4d187c24e89a46645bca6f69b0e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:27:21.483596 327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.351682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42848 (local address 127.10.12.190:36195)
0924 03:27:21.352470 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:21.352480 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:21.385058 (+ 32578us) server_negotiation.cc:407] Connection header received
0924 03:27:21.389992 (+ 4934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:21.389999 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:21.390105 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:21.390194 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:21.396342 (+ 6148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.397590 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.405097 (+ 7507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.406223 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.406352 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:21.471281 (+ 64929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:21.471288 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:21.471294 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:21.471341 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:21.478628 (+ 7287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:21.480213 (+ 1585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:21.480220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:21.480225 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:21.480452 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:21.480941 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:21.480947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:21.480950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:21.481351 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:21.481628 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:21.482905 (+ 1277us) server_negotiation.cc:299] Negotiation successful
0924 03:27:21.483216 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628}
I20260924 03:27:21.479549 373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3aceae61662c43309f33b2eb9b2c4698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4ffbe4d187c24e89a46645bca6f69b0e"
I20260924 03:27:21.488409 372 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4ffbe4d187c24e89a46645bca6f69b0e [term 1 FOLLOWER]: Refusing update from remote peer 3aceae61662c43309f33b2eb9b2c4698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:27:21.492627 373 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4ffbe4d187c24e89a46645bca6f69b0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aceae61662c43309f33b2eb9b2c4698 in term 1.
I20260924 03:27:21.493813 505 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ffbe4d187c24e89a46645bca6f69b0e" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:27:21.495913 478 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65f92a1043d46268f529187fa528e15 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3aceae61662c43309f33b2eb9b2c4698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aceae61662c43309f33b2eb9b2c4698" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36195 } } peers { permanent_uuid: "4ffbe4d187c24e89a46645bca6f69b0e" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } } peers { permanent_uuid: "e65f92a1043d46268f529187fa528e15" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } } }
I20260924 03:27:21.496224 478 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65f92a1043d46268f529187fa528e15 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:27:21.496783 492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3aceae61662c43309f33b2eb9b2c4698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aceae61662c43309f33b2eb9b2c4698" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36195 } } peers { permanent_uuid: "4ffbe4d187c24e89a46645bca6f69b0e" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } } peers { permanent_uuid: "e65f92a1043d46268f529187fa528e15" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } } }
I20260924 03:27:21.497157 492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698 [sys.catalog]: This master's current role is: LEADER
I20260924 03:27:21.498518 513 catalog_manager.cc:1347] Generated new cluster ID: 24460ac8565640a68e1c8275bf6c51f9
I20260924 03:27:21.498622 513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:27:21.517593 513 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:27:21.519517 513 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:27:21.538143 513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698: Generated new TSK 0
I20260924 03:27:21.539456 513 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:27:21.544792 519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ffbe4d187c24e89a46645bca6f69b0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3aceae61662c43309f33b2eb9b2c4698. Latest consensus state: current_term: 1 leader_uuid: "3aceae61662c43309f33b2eb9b2c4698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aceae61662c43309f33b2eb9b2c4698" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36195 } } peers { permanent_uuid: "4ffbe4d187c24e89a46645bca6f69b0e" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } } peers { permanent_uuid: "e65f92a1043d46268f529187fa528e15" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } } }
I20260924 03:27:21.545176 519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ffbe4d187c24e89a46645bca6f69b0e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:27:21.545608 505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3aceae61662c43309f33b2eb9b2c4698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aceae61662c43309f33b2eb9b2c4698" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36195 } } peers { permanent_uuid: "4ffbe4d187c24e89a46645bca6f69b0e" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } } peers { permanent_uuid: "e65f92a1043d46268f529187fa528e15" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } } }
I20260924 03:27:21.545905 505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aceae61662c43309f33b2eb9b2c4698 [sys.catalog]: This master's current role is: LEADER
I20260924 03:27:21.545712 528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ffbe4d187c24e89a46645bca6f69b0e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3aceae61662c43309f33b2eb9b2c4698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aceae61662c43309f33b2eb9b2c4698" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 36195 } } peers { permanent_uuid: "4ffbe4d187c24e89a46645bca6f69b0e" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 32993 } } peers { permanent_uuid: "e65f92a1043d46268f529187fa528e15" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 41875 } } }
I20260924 03:27:21.546228 528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ffbe4d187c24e89a46645bca6f69b0e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:27:21.583657 537 catalog_manager.cc:1259] Loaded cluster ID: 24460ac8565640a68e1c8275bf6c51f9
I20260924 03:27:21.583782 537 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4ffbe4d187c24e89a46645bca6f69b0e: loading cluster ID for follower catalog manager: success
I20260924 03:27:21.587644 537 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4ffbe4d187c24e89a46645bca6f69b0e: acquiring CA information for follower catalog manager: success
I20260924 03:27:21.590430 537 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4ffbe4d187c24e89a46645bca6f69b0e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 03:27:21.706503 327 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:27:21.692961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42864 (local address 127.10.12.190:36195)
0924 03:27:21.693199 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:21.693205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:21.703874 (+ 10669us) server_negotiation.cc:407] Connection header received
0924 03:27:21.704362 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:21.704369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:21.704498 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:21.704593 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:21.706219 (+ 1626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42864: BlockingRecv error: recv got EOF from 127.0.0.1:42864 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 03:27:21.709303 403 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:27:21.695178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48296 (local address 127.10.12.189:32993)
0924 03:27:21.695390 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:21.695398 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:21.707447 (+ 12049us) server_negotiation.cc:407] Connection header received
0924 03:27:21.707755 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:21.707765 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:21.707879 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:21.707981 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:21.709015 (+ 1034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48296: BlockingRecv error: recv got EOF from 127.0.0.1:48296 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 03:27:21.717998 475 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:27:21.705492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41518 (local address 127.10.12.188:41875)
0924 03:27:21.705711 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:21.705718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:21.710011 (+ 4293us) server_negotiation.cc:407] Connection header received
0924 03:27:21.716417 (+ 6406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:21.716424 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:21.716515 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:21.716596 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:21.717711 (+ 1115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41518: BlockingRecv error: recv got EOF from 127.0.0.1:41518 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:27:21.767762 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.750472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48304 (local address 127.10.12.189:32993)
0924 03:27:21.750666 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:21.750675 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:21.761233 (+ 10558us) server_negotiation.cc:407] Connection header received
0924 03:27:21.761523 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:21.761529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:21.761751 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:21.761901 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:27:21.762383 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.763155 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.765180 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.765790 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.765925 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:21.766531 (+ 606us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:27:21.767070 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:27:21.767219 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:21.767315 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 03:27:21.767418 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 03:27:21.814740 475 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:27:21.798411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41534 (local address 127.10.12.188:41875)
0924 03:27:21.798638 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:21.798645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:21.812644 (+ 13999us) server_negotiation.cc:407] Connection header received
0924 03:27:21.812876 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:21.812883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:21.813032 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:21.813133 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:21.814479 (+ 1346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41534: BlockingRecv error: recv got EOF from 127.0.0.1:41534 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 03:27:21.868871 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.851169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48320 (local address 127.10.12.189:32993)
0924 03:27:21.851354 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:21.851361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:21.861893 (+ 10532us) server_negotiation.cc:407] Connection header received
0924 03:27:21.862434 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:21.862440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:21.862544 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:21.862605 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:27:21.863250 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.864437 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.866590 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.867278 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.867389 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:21.868011 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:27:21.868165 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:27:21.868225 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:21.868468 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 03:27:21.868564 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 03:27:21.878042 475 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:27:21.875767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41536 (local address 127.10.12.188:41875)
0924 03:27:21.876005 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:21.876013 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:21.876566 (+ 553us) server_negotiation.cc:407] Connection header received
0924 03:27:21.876763 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:21.876769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:21.876858 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:21.876947 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:21.877807 (+ 860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41536: BlockingRecv error: recv got EOF from 127.0.0.1:41536 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:27:21.889345 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.883065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48332 (local address 127.10.12.189:32993)
0924 03:27:21.883230 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:21.883236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:21.884500 (+ 1264us) server_negotiation.cc:407] Connection header received
0924 03:27:21.884689 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:21.884694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:21.884772 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:21.884856 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:27:21.885264 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.886063 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.887563 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.888026 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.888154 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:21.888577 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:27:21.888708 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:27:21.888788 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:21.888925 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 03:27:21.889012 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 03:27:21.907367 475 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:27:21.904809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41542 (local address 127.10.12.188:41875)
0924 03:27:21.905005 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:21.905011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:21.905667 (+ 656us) server_negotiation.cc:407] Connection header received
0924 03:27:21.905862 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:21.905868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:21.905967 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:21.906102 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:21.907036 (+ 934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41542: BlockingRecv error: recv got EOF from 127.0.0.1:41542 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 03:27:21.923167 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.916418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48346 (local address 127.10.12.189:32993)
0924 03:27:21.916660 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:21.916667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:21.917212 (+ 545us) server_negotiation.cc:407] Connection header received
0924 03:27:21.917417 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:21.917423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:21.917523 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:21.917632 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:27:21.918066 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.919153 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.921146 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.921825 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.921953 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:21.922436 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:27:21.922581 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:27:21.922667 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:21.922784 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 03:27:21.922871 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
W20260924 03:27:21.936794 475 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:27:21.934031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41558 (local address 127.10.12.188:41875)
0924 03:27:21.934458 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:21.934465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:21.935196 (+ 731us) server_negotiation.cc:407] Connection header received
0924 03:27:21.935439 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:21.935445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:21.935555 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:21.935658 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:21.936560 (+ 902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41558: BlockingRecv error: recv got EOF from 127.0.0.1:41558 (error 108)
Metrics: {"server-negotiator.queue_time_us":231}
I20260924 03:27:21.954330 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.948234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48350 (local address 127.10.12.189:32993)
0924 03:27:21.948427 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:21.948434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:21.948971 (+ 537us) server_negotiation.cc:407] Connection header received
0924 03:27:21.949159 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:21.949164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:21.949239 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:21.949317 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:27:21.949754 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.950570 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.952472 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.952978 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.953070 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:21.953607 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:27:21.953736 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:27:21.953800 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:21.953956 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 03:27:21.954030 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 03:27:21.964108 475 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:27:21.961515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41560 (local address 127.10.12.188:41875)
0924 03:27:21.961769 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:21.961776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:21.962336 (+ 560us) server_negotiation.cc:407] Connection header received
0924 03:27:21.962590 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:21.962598 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:21.962707 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:21.962855 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:21.963795 (+ 940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41560: BlockingRecv error: recv got EOF from 127.0.0.1:41560 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 03:27:21.995203 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:21.989327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48360 (local address 127.10.12.189:32993)
0924 03:27:21.989948 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:21.989955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:21.989978 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:27:21.990229 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:21.990235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:21.990340 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:21.990446 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:27:21.990936 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.991807 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.993351 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:21.993859 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:21.993975 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:21.994434 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:27:21.994554 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:27:21.994629 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:21.994856 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 03:27:21.994952 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470}
W20260924 03:27:22.003645 475 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:27:22.001412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41568 (local address 127.10.12.188:41875)
0924 03:27:22.001648 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:22.001655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:22.001945 (+ 290us) server_negotiation.cc:407] Connection header received
0924 03:27:22.002191 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:22.002196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:22.002279 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:22.002371 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:22.003371 (+ 1000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41568: BlockingRecv error: recv got EOF from 127.0.0.1:41568 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
I20260924 03:27:22.046460 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:22.039752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48368 (local address 127.10.12.189:32993)
0924 03:27:22.040005 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:22.040012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:22.040558 (+ 546us) server_negotiation.cc:407] Connection header received
0924 03:27:22.040846 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:22.040850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:22.040990 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:22.041072 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:27:22.041572 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:22.042359 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:22.044337 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:22.044861 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:22.044985 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:22.045557 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:27:22.045735 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:27:22.045846 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:22.046025 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 03:27:22.046118 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 03:27:22.055269 475 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:27:22.053152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41580 (local address 127.10.12.188:41875)
0924 03:27:22.053358 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:22.053365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:22.053788 (+ 423us) server_negotiation.cc:407] Connection header received
0924 03:27:22.053968 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:22.053974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:22.054078 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:22.054202 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:22.055039 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41580: BlockingRecv error: recv got EOF from 127.0.0.1:41580 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 03:27:22.128304 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:22.123116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48382 (local address 127.10.12.189:32993)
0924 03:27:22.123297 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:22.123367 (+ 70us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:22.123806 (+ 439us) server_negotiation.cc:407] Connection header received
0924 03:27:22.124036 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:22.124041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:22.124113 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:22.124191 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:27:22.124628 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:22.125340 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:22.126672 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:22.127084 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:22.127197 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:22.127590 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:27:22.127701 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:27:22.127775 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:22.127922 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 03:27:22.128034 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 03:27:22.135824 475 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:27:22.133851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41586 (local address 127.10.12.188:41875)
0924 03:27:22.134107 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:22.134114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:22.134445 (+ 331us) server_negotiation.cc:407] Connection header received
0924 03:27:22.134657 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:22.134661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:22.134748 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:22.134832 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:22.135605 (+ 773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41586: BlockingRecv error: recv got EOF from 127.0.0.1:41586 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 03:27:22.215531 489 catalog_manager.cc:1259] Loaded cluster ID: 24460ac8565640a68e1c8275bf6c51f9
I20260924 03:27:22.215659 489 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e65f92a1043d46268f529187fa528e15: loading cluster ID for follower catalog manager: success
I20260924 03:27:22.218381 489 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e65f92a1043d46268f529187fa528e15: acquiring CA information for follower catalog manager: success
I20260924 03:27:22.220376 489 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e65f92a1043d46268f529187fa528e15: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:27:22.272809 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:22.267594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48388 (local address 127.10.12.189:32993)
0924 03:27:22.267848 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:22.267856 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:22.268194 (+ 338us) server_negotiation.cc:407] Connection header received
0924 03:27:22.268393 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:22.268397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:22.268529 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:22.268613 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:27:22.269022 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:22.269729 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:22.271126 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:22.271534 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:22.271646 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:22.272091 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:27:22.272219 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:27:22.272296 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:22.272454 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 03:27:22.272541 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 03:27:22.288205 475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:22.281924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.10.12.188:41875)
0924 03:27:22.282114 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:22.282121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:22.282531 (+ 410us) server_negotiation.cc:407] Connection header received
0924 03:27:22.282943 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:22.282948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:22.283121 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:22.283223 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:27:22.283727 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:22.284473 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:22.285841 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:22.286287 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:22.286402 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:22.286943 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:27:22.287480 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:27:22.287620 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:22.287758 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 03:27:22.287851 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 03:27:22.302030 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:22.295252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48404 (local address 127.10.12.189:32993)
0924 03:27:22.295448 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:22.295455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:22.296110 (+ 655us) server_negotiation.cc:407] Connection header received
0924 03:27:22.296327 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:22.296331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:22.296416 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:22.296500 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:27:22.297085 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:22.298110 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:22.300005 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:22.300665 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:22.300806 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:22.301312 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:27:22.301431 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:27:22.301505 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:22.301665 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 03:27:22.301749 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 03:27:22.304148 475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:22.296978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41600 (local address 127.10.12.188:41875)
0924 03:27:22.297177 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:22.297183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:22.297895 (+ 712us) server_negotiation.cc:407] Connection header received
0924 03:27:22.298061 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:22.298066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:22.298165 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:22.298246 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:27:22.298867 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:22.299935 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:22.302011 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:22.302649 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:22.302762 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:22.303267 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:27:22.303416 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:27:22.303478 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:22.303715 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 03:27:22.303808 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 03:27:22.305415 643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:22.295447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42868 (local address 127.10.12.190:36195)
0924 03:27:22.296052 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:22.296058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:22.296937 (+ 879us) server_negotiation.cc:407] Connection header received
0924 03:27:22.297074 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:22.297080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:22.297323 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:22.297528 (+ 205us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:27:22.300037 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:22.300769 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:22.303040 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:22.303702 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:22.303819 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:22.304473 (+ 654us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:27:22.304803 (+ 330us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:27:22.304923 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:22.305037 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 03:27:22.305123 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":342,"threads_started":1}
I20260924 03:27:22.310662 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 32725
I20260924 03:27:22.312116 319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:27:22.426559 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 32725
W20260924 03:27:22.446161 347 connection.cc:535] server connection from 127.0.0.1:48270 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 03:27:22.446816 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 331
I20260924 03:27:22.448081 393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:27:22.557433 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 331
I20260924 03:27:22.574015 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 405
I20260924 03:27:22.575239 467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:27:22.686357 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 405
2026-09-24T03:27:22Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4329 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:27:22 dist-test-slave-00qs krb5kdc[651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:27:22 dist-test-slave-00qs krb5kdc[651](info): set up 2 sockets
Sep 24 03:27:22 dist-test-slave-00qs krb5kdc[651](info): commencing operation
krb5kdc: starting...
W20260924 03:27:23.732481 10290 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/krb5kdc/test-user.keytab.
Sep 24 03:27:23 dist-test-slave-00qs krb5kdc[651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:27:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:27:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
Entry for principal kudu/127.10.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab.
I20260924 03:27:23.973061 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:44419
--webserver_interface=127.10.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:44419,127.10.12.189:39907,127.10.12.188:35347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:27:24.293752 664 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:27:24.294070 664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:27:24.294168 664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:27:24.294275 664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:27:24.301389 664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:27:24.301472 664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:27:24.301542 664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:27:24.301612 664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:27:24.301697 664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:27:24.311470 664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:44419,127.10.12.189:39907,127.10.12.188:35347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.190.keytab
--principal=kudu/127.10.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.190:44419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:27:24.312747 664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:27:24.313962 664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:27:24.325093 672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:24.325135 670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:24.325182 669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:27:24.326017 664 server_base.cc:1034] running on GCE node
I20260924 03:27:24.326788 664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:27:24.328155 664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:27:24.329366 664 hybrid_clock.cc:648] HybridClock initialized: now 1790220444329299 us; error 41 us; skew 500 ppm
I20260924 03:27:24.329698 664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:27:24 dist-test-slave-00qs krb5kdc[651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220444, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:27:24.332939 664 init.cc:377] Logged in from keytab as kudu/127.10.12.190@KRBTEST.COM (short username kudu)
I20260924 03:27:24.334318 664 webserver.cc:460] Webserver started at http://127.10.12.190:37403/ using document root <none> and password file <none>
I20260924 03:27:24.335027 664 fs_manager.cc:359] Metadata directory not provided
I20260924 03:27:24.335124 664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:27:24.335420 664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:27:24.338301 664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-0/data/instance:
uuid: "6a7d1f42774f4c49b5baf0af7c8ef833"
format_stamp: "Formatted at 2026-09-24 03:27:24 on dist-test-slave-00qs"
server_key: "ba84fd2480f32b558cd991b7f1d017f1"
server_key_iv: "ac44414e77850af1a6bd5965f02ef990"
server_key_version: "encryptionkey@0"
I20260924 03:27:24.339049 664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-0/wal/instance:
uuid: "6a7d1f42774f4c49b5baf0af7c8ef833"
format_stamp: "Formatted at 2026-09-24 03:27:24 on dist-test-slave-00qs"
server_key: "ba84fd2480f32b558cd991b7f1d017f1"
server_key_iv: "ac44414e77850af1a6bd5965f02ef990"
server_key_version: "encryptionkey@0"
I20260924 03:27:24.343976 664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260924 03:27:24.347247 679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:27:24.348502 664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:27:24.348649 664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-0/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-0/wal
uuid: "6a7d1f42774f4c49b5baf0af7c8ef833"
format_stamp: "Formatted at 2026-09-24 03:27:24 on dist-test-slave-00qs"
server_key: "ba84fd2480f32b558cd991b7f1d017f1"
server_key_iv: "ac44414e77850af1a6bd5965f02ef990"
server_key_version: "encryptionkey@0"
I20260924 03:27:24.348836 664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:27:24.376972 664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:27:24.381469 664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:27:24.381743 664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:27:24.399156 664 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.190:44419
I20260924 03:27:24.399178 730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.190:44419 every 8 connection(s)
I20260924 03:27:24.400498 664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-0/data/info.pb
I20260924 03:27:24.404049 731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:27:24.408098 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 664
I20260924 03:27:24.408284 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-0/wal/instance
I20260924 03:27:24.408790 10290 external_mini_cluster.cc:1428] Setting key 90aed70eaad9017fa6f3bb9ddbfa3ddb
I20260924 03:27:24.411543 731 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44419 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:27:24 dist-test-slave-00qs krb5kdc[651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220444, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.12.189@KRBTEST.COM; defaulting to no policy
I20260924 03:27:24.434356 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:24.415053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:44419 (local address 127.0.0.1:47140)
0924 03:27:24.415849 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:24.415898 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 03:27:24.418291 (+ 2393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:27:24.419176 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:27:24.419183 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:27:24.419836 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:27:24.420207 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:24.420225 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:24.421710 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:24.421714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:24.422759 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:24.422771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:24.423691 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:24.423697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:24.423903 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:24.424593 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:27:24.424620 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:27:24.427113 (+ 2493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:27:24.430154 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:24.430167 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:24.430180 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:24.430625 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:24.431172 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:24.431176 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:24.431179 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:24.431345 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:24.432420 (+ 1075us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:27:24.432426 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:27:24.432659 (+ 233us) client_negotiation.cc:769] Sending connection context
0924 03:27:24.432868 (+ 209us) client_negotiation.cc:240] Negotiation successful
0924 03:27:24.433270 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":317,"threads_started":1}
I20260924 03:27:24.434376 734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:24.416144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47140 (local address 127.10.12.190:44419)
0924 03:27:24.416822 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:24.416830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:24.416871 (+ 41us) server_negotiation.cc:407] Connection header received
0924 03:27:24.418678 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:24.418684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:24.418940 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:24.419115 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:24.420395 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:24.421568 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:24.422891 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:24.423441 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:24.423596 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:24.427332 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:24.427340 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:24.427352 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:24.427384 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:24.430021 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:24.430776 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:24.430786 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:24.430789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:24.431021 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:24.431523 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:24.431528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:24.431531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:24.432261 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:24.432470 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:24.433074 (+ 604us) server_negotiation.cc:299] Negotiation successful
0924 03:27:24.433458 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":518,"thread_start_us":384,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.189@KRBTEST.COM" created.
I20260924 03:27:24.437312 731 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:27:24.438534 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:24.438104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:39907 (local address 127.0.0.1:58828)
0924 03:27:24.438272 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:24.438377 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 03:27:24.438796 681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.12.189:39907: connect: Connection refused (error 111)
I20260924 03:27:24.440374 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:24.439923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:39907 (local address 127.0.0.1:58834)
0924 03:27:24.440154 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:24.440220 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260924 03:27:24.440568 731 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.189:39907: Network error: Client connection negotiation failed: client connection to 127.10.12.189:39907: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.189.keytab.
Entry for principal kudu/127.10.12.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.189.keytab.
I20260924 03:27:24.467378 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.189:39907
--webserver_interface=127.10.12.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.189.keytab
--principal=kudu/127.10.12.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:44419,127.10.12.189:39907,127.10.12.188:35347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:27:24.489909 731 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } attempt: 1
I20260924 03:27:24.491140 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:24.490623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:39907 (local address 127.0.0.1:58846)
0924 03:27:24.490840 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:24.490970 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 03:27:24.492137 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:24.491746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:39907 (local address 127.0.0.1:58852)
0924 03:27:24.491885 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:24.491986 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 03:27:24.492331 731 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.189:39907: Network error: Client connection negotiation failed: client connection to 127.10.12.189:39907: connect: Connection refused (error 111)
I20260924 03:27:24.560595 731 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } attempt: 2
I20260924 03:27:24.561832 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:24.561294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:39907 (local address 127.0.0.1:58860)
0924 03:27:24.561555 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:24.561665 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 03:27:24.562841 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:24.562401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:39907 (local address 127.0.0.1:58876)
0924 03:27:24.562598 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:24.562691 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260924 03:27:24.563128 731 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.189:39907: Network error: Client connection negotiation failed: client connection to 127.10.12.189:39907: connect: Connection refused (error 111)
I20260924 03:27:24.654394 731 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } attempt: 3
I20260924 03:27:24.655656 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:24.655131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:39907 (local address 127.0.0.1:58886)
0924 03:27:24.655358 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:24.655487 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 03:27:24.656750 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:24.656349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:39907 (local address 127.0.0.1:58894)
0924 03:27:24.656504 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:24.656594 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 03:27:24.656936 731 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.189:39907: Network error: Client connection negotiation failed: client connection to 127.10.12.189:39907: connect: Connection refused (error 111)
W20260924 03:27:24.782227 738 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:27:24.782545 738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:27:24.782647 738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:27:24.782758 738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:27:24.789947 738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:27:24.790036 738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:27:24.790102 738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:27:24.790174 738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:27:24.790252 738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:27:24.799942 738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:44419,127.10.12.189:39907,127.10.12.188:35347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.189.keytab
--principal=kudu/127.10.12.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.189:39907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.12.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:27:24.800160 731 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } attempt: 4
I20260924 03:27:24.801466 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:24.800790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:39907 (local address 127.0.0.1:58898)
0924 03:27:24.801122 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:24.801224 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260924 03:27:24.801465 738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:27:24.802590 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:24.802168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:39907 (local address 127.0.0.1:58906)
0924 03:27:24.802351 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:24.802447 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.189:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 03:27:24.802812 731 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.189:39907: Network error: Client connection negotiation failed: client connection to 127.10.12.189:39907: connect: Connection refused (error 111)
I20260924 03:27:24.803038 738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:27:24.814045 744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:24.814142 743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:24.814118 746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:27:24.814967 738 server_base.cc:1034] running on GCE node
I20260924 03:27:24.815757 738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:27:24.817183 738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:27:24.818457 738 hybrid_clock.cc:648] HybridClock initialized: now 1790220444818386 us; error 46 us; skew 500 ppm
I20260924 03:27:24.818773 738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:27:24 dist-test-slave-00qs krb5kdc[651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220444, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:27:24.821830 738 init.cc:377] Logged in from keytab as kudu/127.10.12.189@KRBTEST.COM (short username kudu)
I20260924 03:27:24.823287 738 webserver.cc:460] Webserver started at http://127.10.12.189:46655/ using document root <none> and password file <none>
I20260924 03:27:24.824016 738 fs_manager.cc:359] Metadata directory not provided
I20260924 03:27:24.824123 738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:27:24.824424 738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:27:24.827257 738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-1/data/instance:
uuid: "35755e7a027b4ea1853a4038990c9a1c"
format_stamp: "Formatted at 2026-09-24 03:27:24 on dist-test-slave-00qs"
server_key: "11c5fd653ce22d2ea400864f5ef9869d"
server_key_iv: "33929ebfe811a5a5521b5154ab67d652"
server_key_version: "encryptionkey@0"
I20260924 03:27:24.828050 738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-1/wal/instance:
uuid: "35755e7a027b4ea1853a4038990c9a1c"
format_stamp: "Formatted at 2026-09-24 03:27:24 on dist-test-slave-00qs"
server_key: "11c5fd653ce22d2ea400864f5ef9869d"
server_key_iv: "33929ebfe811a5a5521b5154ab67d652"
server_key_version: "encryptionkey@0"
I20260924 03:27:24.832960 738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 03:27:24.836289 753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:27:24.837509 738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:27:24.837664 738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-1/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-1/wal
uuid: "35755e7a027b4ea1853a4038990c9a1c"
format_stamp: "Formatted at 2026-09-24 03:27:24 on dist-test-slave-00qs"
server_key: "11c5fd653ce22d2ea400864f5ef9869d"
server_key_iv: "33929ebfe811a5a5521b5154ab67d652"
server_key_version: "encryptionkey@0"
I20260924 03:27:24.837852 738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:27:24.852532 738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:27:24.856412 738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:27:24.856662 738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:27:24.873714 738 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.189:39907
I20260924 03:27:24.873734 811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.189:39907 every 8 connection(s)
I20260924 03:27:24.874996 738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-1/data/info.pb
I20260924 03:27:24.878477 812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:27:24.880133 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 738
I20260924 03:27:24.880285 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-1/wal/instance
I20260924 03:27:24.880690 10290 external_mini_cluster.cc:1428] Setting key 3befd74f16c807048e2aac6574d3acb7
I20260924 03:27:24.884788 812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44419 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:27:24 dist-test-slave-00qs krb5kdc[651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220444, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.189@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.12.188@KRBTEST.COM; defaulting to no policy
I20260924 03:27:24.904847 734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:24.887404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47150 (local address 127.10.12.190:44419)
0924 03:27:24.887616 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:24.887621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:24.888844 (+ 1223us) server_negotiation.cc:407] Connection header received
0924 03:27:24.890695 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:24.890701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:24.890784 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:24.890891 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:24.892312 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:24.893265 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:24.894690 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:24.895281 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:24.895401 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:24.898791 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:24.898798 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:24.898803 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:24.898845 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:24.902058 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:24.902784 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:24.902789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:24.902791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:24.902910 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:24.903236 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:24.903238 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:24.903240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:24.903579 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:24.903690 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:24.904433 (+ 743us) server_negotiation.cc:299] Negotiation successful
0924 03:27:24.904612 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.12.188@KRBTEST.COM" created.
I20260924 03:27:24.905848 815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:24.887854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:44419 (local address 127.0.0.1:47150)
0924 03:27:24.888668 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:24.888718 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 03:27:24.890444 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:27:24.891079 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:27:24.891087 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:27:24.891591 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:27:24.892158 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:24.892176 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:24.893408 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:24.893412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:24.894555 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:24.894567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:24.895528 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:24.895534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:24.895727 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:24.896397 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:27:24.896424 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:27:24.898557 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:27:24.902206 (+ 3649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:24.902213 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:24.902225 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:24.902666 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:24.902995 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:24.902998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:24.903000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:24.903156 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:24.903685 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:27:24.903690 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:27:24.904064 (+ 374us) client_negotiation.cc:769] Sending connection context
0924 03:27:24.904314 (+ 250us) client_negotiation.cc:240] Negotiation successful
0924 03:27:24.904726 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":317,"threads_started":1}
I20260924 03:27:24.907519 812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:27:24 dist-test-slave-00qs krb5kdc[651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220444, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.189@KRBTEST.COM for kudu/127.10.12.189@KRBTEST.COM
I20260924 03:27:24.921191 815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:24.908349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:39907 (local address 127.0.0.1:58916)
0924 03:27:24.908538 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:24.908554 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:27:24.908638 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:27:24.909541 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:27:24.909545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:27:24.909949 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:27:24.910071 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:24.910080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:24.911421 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:24.911426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:24.912282 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:24.912294 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:24.913050 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:24.913055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:24.913220 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:24.913729 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:27:24.913746 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:27:24.915776 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:27:24.918755 (+ 2979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:24.918760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:24.918763 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:24.919203 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:24.919529 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:24.919534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:24.919538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:24.919678 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:24.920263 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:27:24.920267 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:27:24.920417 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 03:27:24.920539 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 03:27:24.920804 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 03:27:24.921270 816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:24.908545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58916 (local address 127.10.12.189:39907)
0924 03:27:24.909166 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:24.909172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:24.909194 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:27:24.909235 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:24.909240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:24.909406 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:24.909535 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:24.910223 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:24.911297 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:24.912400 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:24.912913 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:24.913025 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:24.916025 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:24.916037 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:24.916050 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:24.916079 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:24.918632 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:24.919325 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:24.919332 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:24.919334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:24.919443 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:24.919763 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:24.919766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:24.919768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:24.920144 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:24.920332 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:24.920696 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 03:27:24.920902 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":358,"threads_started":1}
I20260924 03:27:24.923590 812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:27:24.924755 815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:24.924296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35347 (local address 127.0.0.1:52240)
0924 03:27:24.924500 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:24.924599 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 03:27:24.925021 755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35347: connect: Connection refused (error 111)
I20260924 03:27:24.926493 815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:24.926192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35347 (local address 127.0.0.1:52248)
0924 03:27:24.926311 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:24.926385 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 03:27:24.926697 812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:35347: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35347: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.12.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.188.keytab.
Entry for principal kudu/127.10.12.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.188.keytab.
I20260924 03:27:24.935842 10290 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxoV79t/build/asan/bin/kudu
/tmp/dist-test-taskxoV79t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.188:35347
--webserver_interface=127.10.12.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.188.keytab
--principal=kudu/127.10.12.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.12.148:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.12.190:44419,127.10.12.189:39907,127.10.12.188:35347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:27:24.976006 812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } attempt: 1
I20260924 03:27:24.977174 815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:24.976736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35347 (local address 127.0.0.1:52252)
0924 03:27:24.976926 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:24.977029 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 03:27:24.978106 815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:24.977807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35347 (local address 127.0.0.1:52258)
0924 03:27:24.977917 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:24.978005 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 03:27:24.978307 812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:35347: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35347: connect: Connection refused (error 111)
I20260924 03:27:25.046514 812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } attempt: 2
I20260924 03:27:25.047618 815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.047171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35347 (local address 127.0.0.1:52268)
0924 03:27:25.047349 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:25.047468 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 03:27:25.048578 815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.048248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35347 (local address 127.0.0.1:52270)
0924 03:27:25.048377 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:25.048473 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 03:27:25.048774 812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:35347: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35347: connect: Connection refused (error 111)
I20260924 03:27:25.102097 731 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } attempt: 5
Sep 24 03:27:25 dist-test-slave-00qs krb5kdc[651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220444, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for kudu/127.10.12.189@KRBTEST.COM
I20260924 03:27:25.114918 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.102873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:39907 (local address 127.0.0.1:58932)
0924 03:27:25.103094 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:25.103115 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:27:25.103251 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:27:25.103613 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:27:25.103617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:27:25.104067 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:27:25.104275 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:25.104286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.105233 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.105237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:25.106103 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:25.106115 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.106832 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.106838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:25.107035 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:25.107591 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:27:25.107609 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:27:25.109797 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:27:25.112651 (+ 2854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:25.112657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:25.112662 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:25.113012 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:25.113352 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:25.113357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:25.113361 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:25.113498 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:25.113994 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:27:25.113998 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:27:25.114142 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 03:27:25.114259 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 03:27:25.114581 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 03:27:25.114902 816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.102920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58932 (local address 127.10.12.189:39907)
0924 03:27:25.103109 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:25.103116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:25.103192 (+ 76us) server_negotiation.cc:407] Connection header received
0924 03:27:25.103373 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:25.103377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:25.103467 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:25.103553 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:25.104423 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.105105 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.106254 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.106692 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.106800 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:25.109987 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:25.109992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:25.109995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:25.110027 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:25.112526 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:25.113123 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:25.113127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:25.113129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:25.113248 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:25.113595 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:25.113600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:25.113603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:25.113880 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:25.114018 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:25.114372 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 03:27:25.114591 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 03:27:25.116370 731 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:27:25.117316 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.116960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35347 (local address 127.0.0.1:52274)
0924 03:27:25.117112 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:25.117186 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 03:27:25.118219 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.117909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35347 (local address 127.0.0.1:52278)
0924 03:27:25.118048 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:25.118131 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 03:27:25.118403 731 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:35347: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35347: connect: Connection refused (error 111)
I20260924 03:27:25.140008 812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } attempt: 3
I20260924 03:27:25.141255 815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.140738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35347 (local address 127.0.0.1:52288)
0924 03:27:25.140975 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:25.141074 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 03:27:25.142329 815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.141887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35347 (local address 127.0.0.1:52298)
0924 03:27:25.142113 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:25.142194 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260924 03:27:25.142566 812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:35347: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35347: connect: Connection refused (error 111)
I20260924 03:27:25.169620 731 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } attempt: 1
I20260924 03:27:25.170710 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.170251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35347 (local address 127.0.0.1:52314)
0924 03:27:25.170459 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:25.170552 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 03:27:25.171788 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.171378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35347 (local address 127.0.0.1:52318)
0924 03:27:25.171557 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:25.171656 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 03:27:25.172082 731 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:35347: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35347: connect: Connection refused (error 111)
I20260924 03:27:25.240332 731 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } attempt: 2
I20260924 03:27:25.241560 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.241044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35347 (local address 127.0.0.1:52324)
0924 03:27:25.241280 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:25.241378 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 03:27:25.242571 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.242222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35347 (local address 127.0.0.1:52328)
0924 03:27:25.242354 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:25.242448 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 03:27:25.242785 731 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:35347: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35347: connect: Connection refused (error 111)
W20260924 03:27:25.252553 819 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:27:25.252869 819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:27:25.252967 819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:27:25.253103 819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:27:25.260388 819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:27:25.260471 819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:27:25.260560 819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:27:25.260617 819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:27:25.260691 819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:27:25.270619 819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.12.148:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.12.190:44419,127.10.12.189:39907,127.10.12.188:35347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/krb5kdc/kudu_127.10.12.188.keytab
--principal=kudu/127.10.12.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.12.188:35347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.12.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:27:25.271905 819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:27:25.273159 819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:27:25.284832 825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:25.284809 824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:27:25.284757 827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:27:25.285431 819 server_base.cc:1034] running on GCE node
I20260924 03:27:25.285756 812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } attempt: 4
I20260924 03:27:25.286255 819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:27:25.286744 815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.286292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35347 (local address 127.0.0.1:52334)
0924 03:27:25.286513 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:25.286595 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 03:27:25.287669 815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.287330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35347 (local address 127.0.0.1:52348)
0924 03:27:25.287496 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:25.287564 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 03:27:25.287815 819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20260924 03:27:25.287912 812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.12.188:35347: Network error: Client connection negotiation failed: client connection to 127.10.12.188:35347: connect: Connection refused (error 111)
I20260924 03:27:25.289052 819 hybrid_clock.cc:648] HybridClock initialized: now 1790220445288986 us; error 58 us; skew 500 ppm
I20260924 03:27:25.289327 819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:27:25 dist-test-slave-00qs krb5kdc[651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220445, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:27:25.292407 819 init.cc:377] Logged in from keytab as kudu/127.10.12.188@KRBTEST.COM (short username kudu)
I20260924 03:27:25.293730 819 webserver.cc:460] Webserver started at http://127.10.12.188:35767/ using document root <none> and password file <none>
I20260924 03:27:25.294360 819 fs_manager.cc:359] Metadata directory not provided
I20260924 03:27:25.294445 819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:27:25.294696 819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:27:25.297564 819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-2/data/instance:
uuid: "b38f7fe8f2954e8883b496c12f73fee9"
format_stamp: "Formatted at 2026-09-24 03:27:25 on dist-test-slave-00qs"
server_key: "a7651d9d65e145872695faa96011a924"
server_key_iv: "463789b921e288de97a7069f846ec126"
server_key_version: "encryptionkey@0"
I20260924 03:27:25.298321 819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-2/wal/instance:
uuid: "b38f7fe8f2954e8883b496c12f73fee9"
format_stamp: "Formatted at 2026-09-24 03:27:25 on dist-test-slave-00qs"
server_key: "a7651d9d65e145872695faa96011a924"
server_key_iv: "463789b921e288de97a7069f846ec126"
server_key_version: "encryptionkey@0"
I20260924 03:27:25.303274 819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 03:27:25.306671 834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:27:25.308022 819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:27:25.308159 819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-2/data,/tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-2/wal
uuid: "b38f7fe8f2954e8883b496c12f73fee9"
format_stamp: "Formatted at 2026-09-24 03:27:25 on dist-test-slave-00qs"
server_key: "a7651d9d65e145872695faa96011a924"
server_key_iv: "463789b921e288de97a7069f846ec126"
server_key_version: "encryptionkey@0"
I20260924 03:27:25.308346 819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:27:25.324383 819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:27:25.328284 819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:27:25.328507 819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:27:25.345453 819 rpc_server.cc:307] RPC server started. Bound to: 127.10.12.188:35347
I20260924 03:27:25.345459 885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.12.188:35347 every 8 connection(s)
I20260924 03:27:25.346750 819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-2/data/info.pb
I20260924 03:27:25.348703 10290 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxoV79t/build/asan/bin/kudu as pid 819
I20260924 03:27:25.348886 10290 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxoV79t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790220067268388-10290-0/minicluster-data/master-2/wal/instance
I20260924 03:27:25.348968 731 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } attempt: 3
I20260924 03:27:25.349396 10290 external_mini_cluster.cc:1428] Setting key 8d4f37b74fcb6fad0cbfd0834a3b830e
I20260924 03:27:25.350919 886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:27:25.359493 886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44419 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:27:25 dist-test-slave-00qs krb5kdc[651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220444, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.190@KRBTEST.COM for kudu/127.10.12.188@KRBTEST.COM
Sep 24 03:27:25 dist-test-slave-00qs krb5kdc[651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.188@KRBTEST.COM
I20260924 03:27:25.370281 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.349662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35347 (local address 127.0.0.1:52352)
0924 03:27:25.349969 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:25.349987 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:27:25.350082 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:27:25.354236 (+ 4154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:27:25.354241 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:27:25.354760 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:27:25.354974 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:25.354987 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.357473 (+ 2486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.357478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:25.358935 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:25.358951 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.360002 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.360009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:25.360196 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:25.360774 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:27:25.360798 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:27:25.363142 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:27:25.367466 (+ 4324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:25.367471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:25.367475 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:25.367993 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:25.368450 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:25.368455 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:25.368459 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:25.368603 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:25.369460 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:27:25.369465 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:27:25.369664 (+ 199us) client_negotiation.cc:769] Sending connection context
0924 03:27:25.369787 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 03:27:25.369989 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
Sep 24 03:27:25 dist-test-slave-00qs krb5kdc[651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220445, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.188@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:27:25.372887 889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.350592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52352 (local address 127.10.12.188:35347)
0924 03:27:25.351424 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:25.351439 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:25.351485 (+ 46us) server_negotiation.cc:407] Connection header received
0924 03:27:25.353419 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:25.353439 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:25.353890 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:25.354240 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:25.355904 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.357319 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.359117 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.359806 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.360006 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:25.363469 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:25.363479 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:25.363495 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:25.363534 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:25.367307 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:25.368096 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:25.368104 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:25.368111 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:25.368317 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:25.368708 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:25.368714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:25.368717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:25.369303 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:25.370744 (+ 1441us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:25.371041 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 03:27:25.371431 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1489,"server-negotiator.queue_time_us":596,"spinlock_wait_cycles":12672,"thread_start_us":368,"threads_started":1}
I20260924 03:27:25.372860 890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.351468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52356 (local address 127.10.12.188:35347)
0924 03:27:25.352091 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:25.352097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:25.352124 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:27:25.353428 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:25.353447 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:25.353891 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:25.354241 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:25.355860 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.357383 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.359070 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.359823 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.360099 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:25.364953 (+ 4854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:25.364958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:25.364961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:25.364989 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:25.368937 (+ 3948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:25.369934 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:25.369941 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:25.369944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:25.370106 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:25.370590 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:25.370599 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:25.370602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:25.370954 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:25.371215 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:25.371523 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 03:27:25.371701 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":493,"spinlock_wait_cycles":21248,"thread_start_us":272,"threads_started":1}
I20260924 03:27:25.379648 734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.362712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47154 (local address 127.10.12.190:44419)
0924 03:27:25.362898 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:25.362904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:25.363649 (+ 745us) server_negotiation.cc:407] Connection header received
0924 03:27:25.364005 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:25.364010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:25.364087 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:25.364178 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:25.365794 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.366494 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.368517 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.369187 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.369288 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:25.372972 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:25.372978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:25.372983 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:25.373034 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:25.376678 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:25.377471 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:25.377477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:25.377480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:25.377652 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:25.378073 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:25.378078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:25.378081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:25.378468 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:25.378619 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:25.379054 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 03:27:25.379251 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 03:27:25.379648 891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.362735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.190:44419 (local address 127.0.0.1:47154)
0924 03:27:25.363502 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:25.363555 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 03:27:25.363839 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:27:25.364355 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:27:25.364362 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:27:25.365282 (+ 920us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:27:25.365636 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:25.365650 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.366681 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.366686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:25.368345 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:25.368357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.369356 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.369362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:25.369533 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:25.370028 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:27:25.370048 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:27:25.372784 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:27:25.376812 (+ 4028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:25.376823 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:25.376830 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:25.377353 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:25.377769 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:25.377774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:25.377777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:25.377956 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:25.378595 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:27:25.378602 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:27:25.378764 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 03:27:25.378917 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 03:27:25.379236 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":375,"threads_started":1}
I20260924 03:27:25.380369 731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833: Bootstrap starting.
I20260924 03:27:25.381116 886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:27:25.384532 731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833: Neither blocks nor log segments found. Creating new log.
I20260924 03:27:25.385509 731 log.cc:824] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833: Log is configured to *not* fsync() on all Append() calls
Sep 24 03:27:25 dist-test-slave-00qs krb5kdc[651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.190@KRBTEST.COM
I20260924 03:27:25.388625 731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833: No bootstrap required, opened a new log
Sep 24 03:27:25 dist-test-slave-00qs krb5kdc[651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220445, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.188@KRBTEST.COM for kudu/127.10.12.189@KRBTEST.COM
I20260924 03:27:25.393249 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.376100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47168 (local address 127.10.12.190:44419)
0924 03:27:25.376758 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:25.376767 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:25.376788 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:27:25.376829 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:25.376834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:25.376903 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:25.376993 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:25.378385 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.379660 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.381405 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.382086 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.382224 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:25.386851 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:25.386857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:25.386861 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:25.386902 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:25.390297 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:25.391061 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:25.391066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:25.391070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:25.391224 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:25.391635 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:25.391640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:25.391643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:25.392038 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:25.392308 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:25.392612 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 03:27:25.392863 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":650,"threads_started":1}
I20260924 03:27:25.395578 891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.381845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.189:39907 (local address 127.0.0.1:58944)
0924 03:27:25.382112 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:25.382126 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:27:25.382269 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:27:25.382679 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:27:25.382684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:27:25.383100 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:27:25.383268 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:25.383278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.384593 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.384598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:25.385872 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:25.385888 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.386956 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.386960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:25.387089 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:25.387574 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:27:25.387596 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:27:25.389494 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:27:25.392787 (+ 3293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:25.392792 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:25.392796 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:25.393239 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:25.393771 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:25.393776 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:25.393779 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:25.393938 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:25.394666 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:27:25.394671 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:27:25.394825 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 03:27:25.394966 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 03:27:25.395220 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260924 03:27:25.396520 816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.381823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58944 (local address 127.10.12.189:39907)
0924 03:27:25.381992 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:25.381998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:25.382182 (+ 184us) server_negotiation.cc:407] Connection header received
0924 03:27:25.382431 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:25.382437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:25.382537 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:25.382641 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:25.383430 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.384440 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.386045 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.386662 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.386748 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:25.389671 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:25.389676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:25.389679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:25.389710 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:25.392643 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:25.393509 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:25.393515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:25.393517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:25.393647 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:25.394061 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:25.394067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:25.394070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:25.394546 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:25.394698 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:25.395905 (+ 1207us) server_negotiation.cc:299] Negotiation successful
0924 03:27:25.396155 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 03:27:25.397406 886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:27:25.399060 731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44419 } } peers { permanent_uuid: "35755e7a027b4ea1853a4038990c9a1c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } } peers { permanent_uuid: "b38f7fe8f2954e8883b496c12f73fee9" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } }
I20260924 03:27:25.399461 731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a7d1f42774f4c49b5baf0af7c8ef833, State: Initialized, Role: FOLLOWER
I20260924 03:27:25.399943 731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44419 } } peers { permanent_uuid: "35755e7a027b4ea1853a4038990c9a1c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } } peers { permanent_uuid: "b38f7fe8f2954e8883b496c12f73fee9" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } }
Sep 24 03:27:25 dist-test-slave-00qs krb5kdc[651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.12.189@KRBTEST.COM
Sep 24 03:27:25 dist-test-slave-00qs krb5kdc[651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220445, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.188@KRBTEST.COM for kudu/127.10.12.188@KRBTEST.COM
I20260924 03:27:25.406927 895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44419 } } peers { permanent_uuid: "35755e7a027b4ea1853a4038990c9a1c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } } peers { permanent_uuid: "b38f7fe8f2954e8883b496c12f73fee9" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } } }
I20260924 03:27:25.407357 895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:27:25.409148 731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:27:25.415278 891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.397958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35347 (local address 127.0.0.1:52366)
0924 03:27:25.398184 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:25.398200 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:27:25.398280 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:27:25.398578 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:27:25.398582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:27:25.398989 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:27:25.399165 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:25.399177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.400463 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.400468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:25.401730 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:25.401743 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.403008 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.403013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:25.403178 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:25.403713 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:27:25.403733 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:27:25.408577 (+ 4844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:27:25.412406 (+ 3829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:25.412416 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:25.412420 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:25.413172 (+ 752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:25.413565 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:25.413569 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:25.413572 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:25.413711 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:25.414331 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:27:25.414335 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:27:25.414478 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 03:27:25.414628 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 03:27:25.414934 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260924 03:27:25.416286 890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.398118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52366 (local address 127.10.12.188:35347)
0924 03:27:25.398284 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:25.398290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:25.398307 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:27:25.398348 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:25.398352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:25.398430 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:25.398530 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:25.399307 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.400319 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.402124 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.402721 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.402818 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:25.408770 (+ 5952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:25.408776 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:25.408780 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:25.408812 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:25.411999 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:25.413305 (+ 1306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:25.413310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:25.413313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:25.413453 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:25.413828 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:25.413834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:25.413838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:25.414217 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:25.415661 (+ 1444us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:25.415774 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 03:27:25.415997 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 03:27:25.417094 894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.395170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58954 (local address 127.10.12.189:39907)
0924 03:27:25.395664 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:25.395670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:25.395692 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:27:25.395736 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:25.395741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:25.395805 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:25.395908 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:25.397331 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.398471 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.400473 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.401149 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.401843 (+ 694us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:25.407253 (+ 5410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:25.407261 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:25.407266 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:25.407307 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:25.413403 (+ 6096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:25.414179 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:25.414184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:25.414187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:25.414339 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:25.414768 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:25.414773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:25.414775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:25.415107 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:25.416376 (+ 1269us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:25.416494 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 03:27:25.416714 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":233,"threads_started":1}
I20260924 03:27:25.422231 886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b38f7fe8f2954e8883b496c12f73fee9: Bootstrap starting.
W20260924 03:27:25.423190 906 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:27:25.423375 906 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:27:25.425792 10290 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:27:25.427640 886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b38f7fe8f2954e8883b496c12f73fee9: Neither blocks nor log segments found. Creating new log.
I20260924 03:27:25.428074 706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47168
I20260924 03:27:25.428416 706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 03:27:25.428599 706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:27:25.420775 664 master_runner.cc:186] Error getting master registration for 127.10.12.189:39907: OK, instance_id { permanent_uuid: "35755e7a027b4ea1853a4038990c9a1c" instance_seqno: 1790220444868864 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:27:25.431126 664 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:27:25.431079 706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44419 } } peers { permanent_uuid: "35755e7a027b4ea1853a4038990c9a1c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } } peers { permanent_uuid: "b38f7fe8f2954e8883b496c12f73fee9" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } }
I20260924 03:27:25.432457 886 log.cc:824] T 00000000000000000000000000000000 P b38f7fe8f2954e8883b496c12f73fee9: Log is configured to *not* fsync() on all Append() calls
I20260924 03:27:25.432714 706 leader_election.cc:288] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833 [CANDIDATE]: Term 1 election: Requested vote from peers 35755e7a027b4ea1853a4038990c9a1c (127.10.12.189:39907), b38f7fe8f2954e8883b496c12f73fee9 (127.10.12.188:35347)
I20260924 03:27:25.437757 886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b38f7fe8f2954e8883b496c12f73fee9: No bootstrap required, opened a new log
I20260924 03:27:25.449296 886 raft_consensus.cc:348] T 00000000000000000000000000000000 P b38f7fe8f2954e8883b496c12f73fee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44419 } } peers { permanent_uuid: "35755e7a027b4ea1853a4038990c9a1c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } } peers { permanent_uuid: "b38f7fe8f2954e8883b496c12f73fee9" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } }
I20260924 03:27:25.449819 886 raft_consensus.cc:729] T 00000000000000000000000000000000 P b38f7fe8f2954e8883b496c12f73fee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b38f7fe8f2954e8883b496c12f73fee9, State: Initialized, Role: FOLLOWER
I20260924 03:27:25.450414 886 consensus_queue.cc:260] T 00000000000000000000000000000000 P b38f7fe8f2954e8883b496c12f73fee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44419 } } peers { permanent_uuid: "35755e7a027b4ea1853a4038990c9a1c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } } peers { permanent_uuid: "b38f7fe8f2954e8883b496c12f73fee9" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } }
I20260924 03:27:25.465039 915 sys_catalog.cc:455] T 00000000000000000000000000000000 P b38f7fe8f2954e8883b496c12f73fee9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44419 } } peers { permanent_uuid: "35755e7a027b4ea1853a4038990c9a1c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } } peers { permanent_uuid: "b38f7fe8f2954e8883b496c12f73fee9" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } } }
I20260924 03:27:25.465577 915 sys_catalog.cc:458] T 00000000000000000000000000000000 P b38f7fe8f2954e8883b496c12f73fee9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:27:25.466624 886 sys_catalog.cc:564] T 00000000000000000000000000000000 P b38f7fe8f2954e8883b496c12f73fee9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:27:25.467756 861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b38f7fe8f2954e8883b496c12f73fee9"
I20260924 03:27:25.468312 861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b38f7fe8f2954e8883b496c12f73fee9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:27:25.471808 861 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b38f7fe8f2954e8883b496c12f73fee9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a7d1f42774f4c49b5baf0af7c8ef833 in term 1.
I20260924 03:27:25.472949 681 leader_election.cc:302] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a7d1f42774f4c49b5baf0af7c8ef833, b38f7fe8f2954e8883b496c12f73fee9; no voters:
I20260924 03:27:25.473527 895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:27:25.481720 894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.443315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58960 (local address 127.10.12.189:39907)
0924 03:27:25.443501 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:25.443506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:25.454455 (+ 10949us) server_negotiation.cc:407] Connection header received
0924 03:27:25.456069 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:25.456077 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:25.456174 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:25.456250 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:25.460956 (+ 4706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.462133 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.466320 (+ 4187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.466964 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.467069 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:25.473026 (+ 5957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:25.473035 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:25.473040 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:25.473093 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:25.476822 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:25.478476 (+ 1654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:25.478481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:25.478484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:25.478642 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:25.479381 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:25.479388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:25.479391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:25.479801 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:25.480074 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:25.481067 (+ 993us) server_negotiation.cc:299] Negotiation successful
0924 03:27:25.481334 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 03:27:25.488225 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.442060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47178 (local address 127.10.12.190:44419)
0924 03:27:25.442257 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:25.442263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:25.442884 (+ 621us) server_negotiation.cc:407] Connection header received
0924 03:27:25.443454 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:25.443460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:25.443548 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:25.443624 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:25.448913 (+ 5289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.450023 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.454922 (+ 4899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.455654 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.455771 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:25.463192 (+ 7421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:25.463201 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:25.463210 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:25.463261 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:25.467470 (+ 4209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:25.482030 (+ 14560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:25.482037 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:25.482041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:25.483172 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:25.484871 (+ 1699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:25.484877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:25.484880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:25.485264 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:25.485525 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:25.487524 (+ 1999us) server_negotiation.cc:299] Negotiation successful
0924 03:27:25.487791 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 03:27:25.492271 895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833 [term 1 LEADER]: Becoming Leader. State: Replica: 6a7d1f42774f4c49b5baf0af7c8ef833, State: Running, Role: LEADER
I20260924 03:27:25.493465 895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44419 } } peers { permanent_uuid: "35755e7a027b4ea1853a4038990c9a1c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } } peers { permanent_uuid: "b38f7fe8f2954e8883b496c12f73fee9" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } }
I20260924 03:27:25.503136 890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.455161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52374 (local address 127.10.12.188:35347)
0924 03:27:25.455349 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:25.455355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:25.482618 (+ 27263us) server_negotiation.cc:407] Connection header received
0924 03:27:25.483585 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:25.483591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:25.483742 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:25.483816 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:25.486992 (+ 3176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.488183 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.490106 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.490845 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.490944 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:25.495644 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:25.495652 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:25.495657 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:25.495705 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:25.499547 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:25.500441 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:25.500448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:25.500452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:25.500638 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:25.501087 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:25.501092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:25.501096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:25.501531 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:25.501779 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:25.502341 (+ 562us) server_negotiation.cc:299] Negotiation successful
0924 03:27:25.502626 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 03:27:25.505741 918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a7d1f42774f4c49b5baf0af7c8ef833. Latest consensus state: current_term: 1 leader_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44419 } } peers { permanent_uuid: "35755e7a027b4ea1853a4038990c9a1c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } } peers { permanent_uuid: "b38f7fe8f2954e8883b496c12f73fee9" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } } }
I20260924 03:27:25.506520 918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833 [sys.catalog]: This master's current role is: LEADER
I20260924 03:27:25.510874 921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:27:25.513263 921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:27:25.527580 861 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b38f7fe8f2954e8883b496c12f73fee9 [term 1 FOLLOWER]: Refusing update from remote peer 6a7d1f42774f4c49b5baf0af7c8ef833: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:27:25.529299 918 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38f7fe8f2954e8883b496c12f73fee9" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 03:27:25.548671 931 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b38f7fe8f2954e8883b496c12f73fee9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:27:25.548925 931 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:27:25.535807 819 master_runner.cc:186] Error getting master registration for 127.10.12.189:39907: OK, instance_id { permanent_uuid: "35755e7a027b4ea1853a4038990c9a1c" instance_seqno: 1790220444868864 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:27:25.550386 819 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:27:25.553699 915 sys_catalog.cc:455] T 00000000000000000000000000000000 P b38f7fe8f2954e8883b496c12f73fee9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a7d1f42774f4c49b5baf0af7c8ef833. Latest consensus state: current_term: 1 leader_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44419 } } peers { permanent_uuid: "35755e7a027b4ea1853a4038990c9a1c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } } peers { permanent_uuid: "b38f7fe8f2954e8883b496c12f73fee9" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } } }
I20260924 03:27:25.553936 915 sys_catalog.cc:458] T 00000000000000000000000000000000 P b38f7fe8f2954e8883b496c12f73fee9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:27:25.557010 924 mvcc.cc:205] Tried to move back new op lower bound from 7332742944867020800 to 7332742944751636480. Current Snapshot: MvccSnapshot[applied={T|T < 7332742944867020800}]
I20260924 03:27:25.571043 918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44419 } } peers { permanent_uuid: "35755e7a027b4ea1853a4038990c9a1c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } } peers { permanent_uuid: "b38f7fe8f2954e8883b496c12f73fee9" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } } }
I20260924 03:27:25.571372 918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833 [sys.catalog]: This master's current role is: LEADER
I20260924 03:27:25.572335 921 catalog_manager.cc:1347] Generated new cluster ID: 47b26f45a7c243e0b2f074f618fac92c
I20260924 03:27:25.572448 921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:27:25.574101 915 sys_catalog.cc:455] T 00000000000000000000000000000000 P b38f7fe8f2954e8883b496c12f73fee9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44419 } } peers { permanent_uuid: "35755e7a027b4ea1853a4038990c9a1c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } } peers { permanent_uuid: "b38f7fe8f2954e8883b496c12f73fee9" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } } }
I20260924 03:27:25.574333 915 sys_catalog.cc:458] T 00000000000000000000000000000000 P b38f7fe8f2954e8883b496c12f73fee9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:27:25.587102 812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } attempt: 5
Sep 24 03:27:25 dist-test-slave-00qs krb5kdc[651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220444, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.12.189@KRBTEST.COM for kudu/127.10.12.188@KRBTEST.COM
I20260924 03:27:25.605564 890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.587999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52384 (local address 127.10.12.188:35347)
0924 03:27:25.588143 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:25.588149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:25.588165 (+ 16us) server_negotiation.cc:407] Connection header received
0924 03:27:25.588213 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:25.588217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:25.588281 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:25.588373 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:25.589253 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.590262 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.591939 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.592607 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.592715 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:25.597191 (+ 4476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:27:25.597198 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:27:25.597203 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:27:25.597242 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:27:25.600761 (+ 3519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:25.602204 (+ 1443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:25.602210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:25.602214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:25.602368 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:27:25.603035 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:27:25.603040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:27:25.603043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:27:25.603401 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:27:25.603543 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:25.604404 (+ 861us) server_negotiation.cc:299] Negotiation successful
0924 03:27:25.605168 (+ 764us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 03:27:25.607250 815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:25.587687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.12.188:35347 (local address 127.0.0.1:52384)
0924 03:27:25.587876 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 03:27:25.587896 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:27:25.588038 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:27:25.588400 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:27:25.588405 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:27:25.588918 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:27:25.589124 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:25.589137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.590423 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.590429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:25.591811 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:27:25.591826 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:27:25.592744 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:27:25.592750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:27:25.592957 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:25.593609 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:27:25.593633 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:27:25.596918 (+ 3285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:27:25.601034 (+ 4116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:25.601040 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:25.601044 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:25.601941 (+ 897us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:25.602563 (+ 622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:27:25.602595 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:27:25.602599 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:27:25.602866 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:27:25.603718 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:27:25.603724 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:27:25.603986 (+ 262us) client_negotiation.cc:769] Sending connection context
0924 03:27:25.606064 (+ 2078us) client_negotiation.cc:240] Negotiation successful
0924 03:27:25.606586 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 03:27:25.607705 921 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:27:25.609560 921 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:27:25.613999 812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35755e7a027b4ea1853a4038990c9a1c: Bootstrap starting.
I20260924 03:27:25.620958 812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35755e7a027b4ea1853a4038990c9a1c: Neither blocks nor log segments found. Creating new log.
I20260924 03:27:25.623333 812 log.cc:824] T 00000000000000000000000000000000 P 35755e7a027b4ea1853a4038990c9a1c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:27:25.638504 921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833: Generated new TSK 0
I20260924 03:27:25.639505 812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35755e7a027b4ea1853a4038990c9a1c: No bootstrap required, opened a new log
I20260924 03:27:25.640599 921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:27:25.654177 812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35755e7a027b4ea1853a4038990c9a1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44419 } } peers { permanent_uuid: "35755e7a027b4ea1853a4038990c9a1c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } } peers { permanent_uuid: "b38f7fe8f2954e8883b496c12f73fee9" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } }
I20260924 03:27:25.654554 812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35755e7a027b4ea1853a4038990c9a1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35755e7a027b4ea1853a4038990c9a1c, State: Initialized, Role: FOLLOWER
I20260924 03:27:25.655019 812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35755e7a027b4ea1853a4038990c9a1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44419 } } peers { permanent_uuid: "35755e7a027b4ea1853a4038990c9a1c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } } peers { permanent_uuid: "b38f7fe8f2954e8883b496c12f73fee9" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } }
I20260924 03:27:25.670073 938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35755e7a027b4ea1853a4038990c9a1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44419 } } peers { permanent_uuid: "35755e7a027b4ea1853a4038990c9a1c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } } peers { permanent_uuid: "b38f7fe8f2954e8883b496c12f73fee9" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } } }
I20260924 03:27:25.671418 812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35755e7a027b4ea1853a4038990c9a1c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:27:25.671468 787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "35755e7a027b4ea1853a4038990c9a1c"
I20260924 03:27:25.671904 787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35755e7a027b4ea1853a4038990c9a1c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:27:25.675689 787 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 35755e7a027b4ea1853a4038990c9a1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a7d1f42774f4c49b5baf0af7c8ef833 in term 1.
I20260924 03:27:25.676635 786 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 35755e7a027b4ea1853a4038990c9a1c [term 1 FOLLOWER]: Refusing update from remote peer 6a7d1f42774f4c49b5baf0af7c8ef833: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:27:25.677779 895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35755e7a027b4ea1853a4038990c9a1c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:27:25.685813 938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35755e7a027b4ea1853a4038990c9a1c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:27:25.704104 944 mvcc.cc:205] Tried to move back new op lower bound from 7332742944867020800 to 7332742944751636480. Current Snapshot: MvccSnapshot[applied={T|T < 7332742944867020800}]
I20260924 03:27:25.714044 938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35755e7a027b4ea1853a4038990c9a1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a7d1f42774f4c49b5baf0af7c8ef833. Latest consensus state: current_term: 1 leader_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44419 } } peers { permanent_uuid: "35755e7a027b4ea1853a4038990c9a1c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } } peers { permanent_uuid: "b38f7fe8f2954e8883b496c12f73fee9" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } } }
I20260924 03:27:25.714427 938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35755e7a027b4ea1853a4038990c9a1c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:27:25.714887 942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35755e7a027b4ea1853a4038990c9a1c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44419 } } peers { permanent_uuid: "35755e7a027b4ea1853a4038990c9a1c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } } peers { permanent_uuid: "b38f7fe8f2954e8883b496c12f73fee9" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } } }
I20260924 03:27:25.715250 942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35755e7a027b4ea1853a4038990c9a1c [sys.catalog]: This master's current role is: FOLLOWER
W20260924 03:27:25.803272 892 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:27:25.789666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47192 (local address 127.10.12.190:44419)
0924 03:27:25.790173 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:25.790180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:25.800841 (+ 10661us) server_negotiation.cc:407] Connection header received
0924 03:27:25.801073 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:25.801080 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:25.801199 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:25.801324 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:25.803009 (+ 1685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47192: BlockingRecv error: recv got EOF from 127.0.0.1:47192 (error 108)
Metrics: {"server-negotiator.queue_time_us":360}
W20260924 03:27:25.805529 894 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:27:25.789942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58970 (local address 127.10.12.189:39907)
0924 03:27:25.790339 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:25.790346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:25.803450 (+ 13104us) server_negotiation.cc:407] Connection header received
0924 03:27:25.803527 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:25.803532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:25.803639 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:25.803736 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:25.805279 (+ 1543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58970: BlockingRecv error: recv got EOF from 127.0.0.1:58970 (error 108)
Metrics: {"server-negotiator.queue_time_us":258}
I20260924 03:27:25.806916 962 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260924 03:27:25.809116 890 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:27:25.801548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52390 (local address 127.10.12.188:35347)
0924 03:27:25.801739 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:25.801745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:25.805328 (+ 3583us) server_negotiation.cc:407] Connection header received
0924 03:27:25.806522 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:25.806529 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:25.806617 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:25.806686 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:25.808856 (+ 2170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52390: BlockingRecv error: recv got EOF from 127.0.0.1:52390 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 03:27:25.822068 894 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:27:25.818582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58972 (local address 127.10.12.189:39907)
0924 03:27:25.818756 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:25.818763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:25.820468 (+ 1705us) server_negotiation.cc:407] Connection header received
0924 03:27:25.820701 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:25.820707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:25.820840 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:25.820940 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:25.821849 (+ 909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58972: BlockingRecv error: recv got EOF from 127.0.0.1:58972 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 03:27:25.834040 894 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:27:25.830307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58978 (local address 127.10.12.189:39907)
0924 03:27:25.830521 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:25.830528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:25.832508 (+ 1980us) server_negotiation.cc:407] Connection header received
0924 03:27:25.832697 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:25.832703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:25.832800 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:25.832916 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:25.833782 (+ 866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58978: BlockingRecv error: recv got EOF from 127.0.0.1:58978 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 03:27:25.845901 894 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:27:25.839154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58984 (local address 127.10.12.189:39907)
0924 03:27:25.839317 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:25.839323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:25.844499 (+ 5176us) server_negotiation.cc:407] Connection header received
0924 03:27:25.844666 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:25.844672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:25.844765 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:25.844873 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:25.845677 (+ 804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58984: BlockingRecv error: recv got EOF from 127.0.0.1:58984 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 03:27:25.855391 894 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:27:25.852888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58998 (local address 127.10.12.189:39907)
0924 03:27:25.853128 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:25.853134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:25.853761 (+ 627us) server_negotiation.cc:407] Connection header received
0924 03:27:25.854052 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:25.854058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:25.854148 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:25.854223 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:25.855151 (+ 928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58998: BlockingRecv error: recv got EOF from 127.0.0.1:58998 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
W20260924 03:27:25.869943 894 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:27:25.867608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59012 (local address 127.10.12.189:39907)
0924 03:27:25.867787 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:25.867794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:25.868458 (+ 664us) server_negotiation.cc:407] Connection header received
0924 03:27:25.868670 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:25.868675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:25.868782 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:25.868894 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:25.869727 (+ 833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59012: BlockingRecv error: recv got EOF from 127.0.0.1:59012 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 03:27:25.907503 894 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:27:25.905277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59028 (local address 127.10.12.189:39907)
0924 03:27:25.905485 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:25.905491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:25.905993 (+ 502us) server_negotiation.cc:407] Connection header received
0924 03:27:25.906193 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:25.906198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:25.906298 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:25.906415 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:25.907251 (+ 836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59028: BlockingRecv error: recv got EOF from 127.0.0.1:59028 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260924 03:27:25.945960 894 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:27:25.943610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59042 (local address 127.10.12.189:39907)
0924 03:27:25.943818 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:25.943825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:25.944389 (+ 564us) server_negotiation.cc:407] Connection header received
0924 03:27:25.944595 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:25.944601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:25.944704 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:25.944820 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:25.945709 (+ 889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59042: BlockingRecv error: recv got EOF from 127.0.0.1:59042 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 03:27:26.015295 894 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:27:26.013383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59046 (local address 127.10.12.189:39907)
0924 03:27:26.013575 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:26.013582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:26.013908 (+ 326us) server_negotiation.cc:407] Connection header received
0924 03:27:26.014080 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:26.014086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:26.014224 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:26.014339 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:26.015043 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59046: BlockingRecv error: recv got EOF from 127.0.0.1:59046 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 03:27:26.148561 894 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:27:26.146687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59050 (local address 127.10.12.189:39907)
0924 03:27:26.146865 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:26.146870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:26.147192 (+ 322us) server_negotiation.cc:407] Connection header received
0924 03:27:26.147412 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:26.147417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:26.147502 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:26.147597 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:26.148343 (+ 746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59050: BlockingRecv error: recv got EOF from 127.0.0.1:59050 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 03:27:26.224717 895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d1f42774f4c49b5baf0af7c8ef833" member_type: VOTER last_known_addr { host: "127.10.12.190" port: 44419 } } peers { permanent_uuid: "35755e7a027b4ea1853a4038990c9a1c" member_type: VOTER last_known_addr { host: "127.10.12.189" port: 39907 } } peers { permanent_uuid: "b38f7fe8f2954e8883b496c12f73fee9" member_type: VOTER last_known_addr { host: "127.10.12.188" port: 35347 } } }
I20260924 03:27:26.225059 895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a7d1f42774f4c49b5baf0af7c8ef833 [sys.catalog]: This master's current role is: LEADER
W20260924 03:27:26.413012 894 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:27:26.411018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59064 (local address 127.10.12.189:39907)
0924 03:27:26.411212 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:26.411220 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:26.411571 (+ 351us) server_negotiation.cc:407] Connection header received
0924 03:27:26.411846 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:26.411851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:26.411931 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:26.412044 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:27:26.412767 (+ 723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59064: BlockingRecv error: recv got EOF from 127.0.0.1:59064 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:27:26.551014 931 catalog_manager.cc:1259] Loaded cluster ID: 47b26f45a7c243e0b2f074f618fac92c
I20260924 03:27:26.551112 931 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b38f7fe8f2954e8883b496c12f73fee9: loading cluster ID for follower catalog manager: success
I20260924 03:27:26.553825 931 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b38f7fe8f2954e8883b496c12f73fee9: acquiring CA information for follower catalog manager: success
I20260924 03:27:26.555572 931 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b38f7fe8f2954e8883b496c12f73fee9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:27:26.811098 962 catalog_manager.cc:1259] Loaded cluster ID: 47b26f45a7c243e0b2f074f618fac92c
I20260924 03:27:26.811223 962 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 35755e7a027b4ea1853a4038990c9a1c: loading cluster ID for follower catalog manager: success
I20260924 03:27:26.814057 962 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 35755e7a027b4ea1853a4038990c9a1c: acquiring CA information for follower catalog manager: success
I20260924 03:27:26.815812 962 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 35755e7a027b4ea1853a4038990c9a1c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:27:26.943293 1023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:26.936368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59076 (local address 127.10.12.189:39907)
0924 03:27:26.937172 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:26.937179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:26.937202 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:27:26.937284 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:26.937289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:26.937497 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:26.937629 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:27:26.938027 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:26.939204 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:26.940526 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:26.941223 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:26.941345 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:26.941918 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:27:26.942631 (+ 713us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:27:26.942775 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:26.942881 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 03:27:26.942970 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":527,"threads_started":1}
I20260924 03:27:26.955814 1029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:26.950029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52398 (local address 127.10.12.188:35347)
0924 03:27:26.950614 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:26.950620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:26.950639 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:27:26.950855 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:26.950860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:26.951032 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:26.951177 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:27:26.951574 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:26.952364 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:26.953724 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:26.954108 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:26.954210 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:26.954709 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:27:26.955235 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:27:26.955374 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:26.955448 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 03:27:26.955531 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":361,"threads_started":1}
I20260924 03:27:26.990937 1029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:26.976002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52412 (local address 127.10.12.188:35347)
0924 03:27:26.976162 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:26.976167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:26.984299 (+ 8132us) server_negotiation.cc:407] Connection header received
0924 03:27:26.984450 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:26.984457 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:26.984567 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:26.984657 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:27:26.985843 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:26.986691 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:26.988903 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:26.989524 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:26.989620 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:26.990083 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:27:26.990259 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:27:26.990339 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:26.990566 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 03:27:26.990654 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 03:27:26.993116 1036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:26.968099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47194 (local address 127.10.12.190:44419)
0924 03:27:26.984740 (+ 16641us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:26.984745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:26.984763 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:27:26.984862 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:26.984865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:26.985062 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:26.985252 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:27:26.985898 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:26.987715 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:26.989997 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:26.990613 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:26.990718 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:26.992081 (+ 1363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:27:26.992530 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:27:26.992638 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:26.992775 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 03:27:26.992859 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16522,"thread_start_us":16419,"threads_started":1}
I20260924 03:27:26.993162 1023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:27:26.967992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59078 (local address 127.10.12.189:39907)
0924 03:27:26.968195 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 03:27:26.968202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:27:26.983859 (+ 15657us) server_negotiation.cc:407] Connection header received
0924 03:27:26.984792 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:27:26.984799 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:27:26.984896 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:27:26.984984 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:27:26.986183 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:26.987289 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:26.990429 (+ 3140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:27:26.991082 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:27:26.991178 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:27:26.991979 (+ 801us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:27:26.992102 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:27:26.992198 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 03:27:26.992803 (+ 605us) server_negotiation.cc:299] Negotiation successful
0924 03:27:26.992893 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 03:27:27.001888 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 664
I20260924 03:27:27.002931 726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:27:27.112599 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 664
I20260924 03:27:27.140087 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 738
I20260924 03:27:27.141243 807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:27:27.258535 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 738
I20260924 03:27:27.274782 10290 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxoV79t/build/asan/bin/kudu pid 819
I20260924 03:27:27.275846 881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:27:27.381793 10290 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxoV79t/build/asan/bin/kudu with pid 819
2026-09-24T03:27:27Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4693 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9023 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (380071 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST